The Shells, The Shells, Esmerelda!

You know, given his thpeeth impedimenth, you’d have thought Quasimodo would have found a girlfriend with an easier name, maybe a Betty or an Anna (probably not a Thuthan).  Anyway, that has nothing to do with the subject of today’s content-free babblings, which is the crossover point between shells and environments.

Jeremy Zawodny writes on using the MySQL client like a shell, according to this Forever Geek post.  Now, having rattled on about using interactive Python as an environment for working on MySQL a while back, I thought this could be interesting, until I found that it was just a short blog on aliasing the “cd” and “ls” commands to “use” and “show tables”.  Sort of useful, I guess.

But it got me thinking about shells.  Rather than attempt to gradually retrofit mysql with the basic commands that a shell has, why not take a programmable environment like, oh, I dunno, let me take something at random, say, Python and extend that to hook better into MySQL, or whatever other environment you happen to be in.

There are a few Python shells around, such as IPython and PyCrust or the extremely bashalike PySH (which I like to pronounce to rhyme with fish, just because I can).  I took a look at each one, and found that the whole subject is packed with Interesting Questions.

First off, what’s a Python Shell?  To some people, it’s an interactive Python environment.  To others, it’s something more like bash or cmd, an environment for interacting with an operating system, but which has a degree of programmability (in Python, naturally).  My definition would be the latter – I like command-line shells.  To me, bash is Unix, in the sense that the underlying OS is rather less important than the pipe/small commands/vi nature of the environment, which is why I like cygwin.

Second in the list of Interesting Things To Ponder, it seems to me, is that of the context within which commands are executed.  For instance, all OS shells that I have ever used have a notion of a current working directory, whether this is set explicitly by cd or as some sort of default value for command expansion, as in VMS’s set def.  The mysql client has a notion of the current database which is arguably similar; also it has a connection to a given database server, specified at startup.  These are its context, thus to integrate MySQL into a Python shell, we need to extend the “current context” to include both the server and the current database.

But context-dependence is a double-edged sword.  As well as the working context within which Stuff Happens, there’s the deep and complex question of how a given line of input is interpreted, when there are multiple contexts in which the various elements of it can be assigned semantics.

But that’s putting it too formally.  What I did next was to write some commands that might work in my ideal shell.  First off, I want the ability to do something like:

for x in `ls *.py`: cp x x.replace('.py',.py.sav')

For now, ignore the mix of bash and Python; I want to be able to use ls to get me a list of names and use that as the sequence over which to iterate a for loop.  Then I want to be able to apply Python string operations to the variables to do clever stuff with names, and I want the operating system commands, like cp to be invoked for me.

Here’s another:

for a in "Table1 Table2".split():
  rows = select * from a
  print "%s has %d rows" % (a,len(rows))

After all, the original article that sparked this off was about extending MySQL, so why shouldn’t I be able to mix in SQL commands as well as OS stuff?

One way to do this is to try adding new keywords to Python, but that way madness lies; also that way involves C programming, and I do this sort of stuff for fun, so why drop into another language?  No, there had to be a way to do this in pure Python.

Have you ever had a problem work like a catalyst in your head?  You know the way that grit acts on an oyster, provoking the formation of a pearl, or a seed crystal dropped into a solution can kick off the rapid growth of a bigger structure?  This was like that – over two nights I lost sleep whilst my brain tickled away at the problem.  I dreamed in code, something that hasn’t happened in a long time.  Finally, after two days of nibbling away at the issue without going near a keyboard, I thought I had a breakthrough.  Take a look at the second example above.  How does the interpreter know that the “select” is a SQL expression?  That the “*” is to be passed to the SQL interpreter, where a Unix-esque shell would glob it in place.  I decided that the solution had two parts.  The first is disambiguation: the context in which words are to be interpreted has to be more explicit.  The second is that the shell should be a preprocessor – it would rewrite the input lines so that the Python interpreter always executes pure Python.

The next question – what was an appropriate syntax to use?  I chose to use the backtick, “`”, for these reason:

  • It’s used in Unix shells for a roughly similar purpose – to bracket a command that’s to be executed in a subshell and its output captured, rather like that first example command above.
  • It’s used in Python only as syntactic sugar – one can replace `a` with str(a), so a Python in which the meaning of “ has been overridden loses nothing.

Thus the shell can take any sequence of characters within backticks and rewrite it to be legal Python that will have the desired effect.
Let’s go back to that example:

for x in `ls *.py`

It’s more-or-less equivalent to:

for x in os.popen("ls *.py")[1].readlines():

Python that has the desired effect.

However, I also want to be able to mix in SQL commands, so the shell has to be able to support multiple different contexts in which stuff can be understood.  I also want to be able to add new contexts as I feel like it; choosing a new alternative to backticks for each one would get difficult very fast, so I chose to stay with backticks and disambiguate some other way.

Finally, I chose a name for the shell: Quasi.  This is because:

  • It let me make terrible jokes; when asked how I thought of it, I could reply ‘I had a hunch…’
  • I liked this title of this entry too much not to make the link
  • Nobody will ever be sure how to pronounce it – does the A rhyme with “day” and the I with “eye”, or does the whole rhyme with “Ozzy”?

And after a weekend during which I wrote altogether too much code to be healthy, version 0.1 exists.  You can download the source, if you really want to, here..

So what can it do?  Here are some examples of commands that really work:

>>>#Get a list of all Python files in the current directory and copy them to backup versions.
>>>for x in `os ls *.py`: `cp $x $(x.replace('.py','.py.sav'))`
>>>#connect to a MySQL server.  The keyword "sql" at the start of the line
>>>#is syntactic sugar that lets us leave off the `` ticks.
>>>sql connect host=myHost, user=MyUser, passwd=MyPassword, db=myDb
>>>#do a query and capture the results
>>>table = 'Sample'
>>>rows = `sql select * from $table`
>>>rows
(('Some', 'All'), ('Here', 'There'), ('A', 'B'))

There are three contexts:

  • The SQL context, marked by the keyword “sql” at the start of the line or after the opening backtick (or the character “&”, so `& select * from Sample` is legal).
  • The OS context, marked by the keyword “os”.  OS commands return their output (captured from popen) as a list of strings.
  • The shell context, marked by the keyword “shell” or “!”.  Shell commands return their status, but not their output, so you can do `shell vi $x` to edit the file whose name is in x and it’ll work.

There’s an extra trick for the OS context – if you put a variable at the start, before a “|” (pipe), then that variable will be printed to the stdin of the process.  It handles lists appropriately.
Variable substitution is done with the syntax $ (where the end of the name is recognized sensibly) or $(), where everything between the parentheses is evaluated by Python.  So the following are all legal:

$a
$(a)
$(a.lower())
$(a.strip().lower())
$(getattr(a,"test",None))
$(os.system(a.strip()+".exe")[1].readlines()))

Those of a curious disposition may like to type “trace 3” as their first command – you can see the Python that your command expands to.

Next entry – how the damn thing works, and musings on the interesting nature of shells that have built-in interpreters… code that executes in two different namespaces.

Or not.

13 thoughts on “The Shells, The Shells, Esmerelda!

  1. in quasi.QuasiOs.execute, around 451, the has_Popen3 case: Popen3.fromchild is a file, like tochild, not a callable; likewise (at least in 2.3) Popen3 doesn’t have a close() method, using p.fromchild.close() on the next line makes more sense.

    finally, line 484, the #Trim any trailing newlines/returns from the result lines
    comment and subsequent output=map… should be outdented by one, or (again in the has_Popen3 case) output is never set. With these changes, os ls *.py works (on OSX 10.3.4, python 2.3); I haven’t tried much of the rest yet.

    • Many thanks, Mark – both of those fixed. Mea maxima culpa for not testing everytime on Unix, even with a drag-and-drop WinSCP running all the time. 🙂
      regards
      b

      • repr not str

        >> It’s used in Python only as syntactic sugar – one can replace `a`
        >> with str(a), so a Python in which the meaning of “ has been
        >> overridden loses nothing.

        ITYM “repr(a)”

      • Re: repr not str

        ITID, and having gone on about the subject of str vs. repr in another blog entry, I should have checked 🙂 Interesting enough, the difference between str(a) and repr(a) (where a is a string) is crucial in Quasi, since there’s lot of code that generates strings to be evaluated by a Python interpreter later, and escaping of quotes and other characters is pretty crucial.
        regards
        ben

  2. ipython

    You might want to check out the latest ipython from cvs, download the rc1 of the next release from:

    http://ipython.scipy.org/dist/testing/

    or just wait for the release, possibly someday next week. When executed with “ipython -p pysh”, it functions quite comfortably as a system shell.

    # Ville Vainio

    • Re: ipython

      I might do that, but Quasi is so interesting that I’d do it just for the fun of it! Also – I like the ability to mix up OS, Python and SQL in Quasi-style. I’m considering an SSH context and some others also… contexts are an interesting idea, and I’m going to see where it all goes.
      thanks and regards
      b

      • Re: ipython

        I might do that, but Quasi is so interesting that I’d do it just for the fun of it!

        No doubt – in any case, ipython is going to be tackling issues like this soon as well, refactoring will start after a couple of releases and context issues & such will be resolved.

        So far, I’ve thought that it would be handy if the contexts were resolved from the name of the function; i.e. in the “function dictionary”, key “cp” could have a reference to a shared StdOsCommand object, which would have various attributes instructing the interpreter environment on command completion, arg preprocessing, etc…

        But that’s not on table yet, I’ll check back on Quasi when the time is ripe. But be sure to check out ipython once the release is made, perhaps some of the Quasi stuff is going to be useful there too, and there is already lots of handy readline/coloring/other_comfort_related plumbing that is going to be needed anyway. Python-based system shell is way overdue, esp. as bash doesn’t feel very natural on Wintendo because of the cygwin wrapping.

      • Re: ipython

        Shortcuts like “cp” automatically invoking an OS (or more likely, Shell) context are in the pipeline – my current preference is to allow these via some form of aliasing, so that different users can set up different sets of them; after all, dyed-in-the-wool-Windows users aren’t going to want an “ls”, they’ll want a “dir” (and disambiguating that from Python’s dir() is very interesting).
        I was thinking about syntax colouring this morning (during a two-hour drive at 6am); I’m going to take a look at SPE, PythonWin et al to see how they do it, so ipython is now also on that list. I’m not proud, I’ll steal ideas from anyone 🙂 Syntax colouring would be especially useful for contexts. There’s a refactoring on the list to rework the main parser to deal with input character-by-character to help support that… but then I’ll need to build a noddy wxPython framework for it to live in too.
        regards
        b

      • Re: ipython

        Another take on this is REXX, which has a way of executing external commands, and an ADDRESS statement to set the context. I can’t recall the details (it’s been a while since I looked at REXX) but it sounds very similar in principle to what you’re doing.

      • Re: ipython

        I use the sh from MSYS and feel wuite happy with it, and with the associated sed, awk and so on

      • Re: ipython

        Disclaimer: I am NOT trying to discourage you in _any_ way of developing Quasi for fun. But I’d just like to mention that ipython is already fairly mature, it has a wide user base, but it sorely needs more developers. So I’d be _really_ happy to have a few more helping hands here.

        I’m just making this comment to see if I can snag you for the benefit of having an even better tool that could combine your ideas with mine (and those of many others). While you may find developing for scratch more fun, on the flip side, joining efforts would let you leverage ipython’s existing codebase (~14k lines).

        Anyway, good luck with Quasi, and feel free to contact me at fperez@colorado.edu (or even better, on the ipython lists) if you are interested at all.

        Best regards,

        Fernando Perez — ipython author.

      • Re: ipython

        You make a good point, but in fact the two aren’t mutually exclusive. Only after putting some of the basic ideas in Quasi into practice do I now understand how they need to be done – automatic disambiguation is an Interesting problem precisely because it’s *hard*. So what I could do is take stuff that’s been proven in Quasi and port it over. Anyway, I’ll get hold of ipython today and take a look.
        Thanks
        b

  3. wow

    You have no idea how many times I’ve started typing “for f in (ls /home/sjames)…” only to realize that I don’t live in my dreamworld. Now I can! I will definately have to play with this.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s