You have available to you a sample lexer built with JLex. In the great tradition of
building complex systems, I suggest you start building your own lexer by
incremental modification of this one. Do it incrementally - one change at a time!
There are two files provided one called sym.java
which is a class that specifies what the token (lexeme) values are. There is also the file
sample.lex which is the input into JLex.
Getting it together
Before you start working with JLex, you need to set some things up. Firstly, you need
to make sure JLex is available to you. The JLex system needs to appear within a directory
called (wait for it!) JLex on your Java CLASSPATH. If you are using the JLex
system on the Department's computers, this should already be done for you.
If you are using JLex at home, you'll need to set it up. If you have messed around with
your CLASSPATH environment variable in your account on the Departmental computers, you may
also need to set it up again.
Running Jlex
You will need to create a directory called "compiler" and copy the sym.java file there. Compile that file in the compiler
directory (it is in a Java package called compiler).
You are now ready to run Jlex. Move back to the directory that contains the sample.lex file.
Running Jlex is a breeze. You just feed in the input file (sample.lex is the one we
provide). Like this:
[kevin@akurra ccp]$ java JLex/Main sample.lex
Processing first section -- user code.
Processing second section -- JLex declarations.
Processing third section -- lexical rules.
Creating NFA machine representation.
NFA comprised of 216 states.
Working on character classes.:::::::::::::::::......::..:..:...................::.............::::::::.:::::::..:..::.
NFA has 43 distinct character classes.
Creating DFA transition table.
Working on DFA states...................................................................................................
Minimizing DFA transition table.
69 states after removal of redundant states.
Outputting lexical analyzer code.
This is the output I get from sample.lex.
The result of running Jlex on sample.lex is a file called sample.lex.java. We now need
to run that through the Java compiler before we can run the program!
javac sample.lex.java
... is all you need to do. If you get errors at this stage, you need to go back and
edit the .lex file and then feed that back through Jlex before re-compiling.
What you now have is the sample class compiled and ready to go. Shown below is a sample
run. Make sure you use a capital "S" in sample!
[kevin@akurra ccp]$ java Sample
foo
Token #25: foo (line 0, character pos0:3 )
foo := foo + 1 ;
Token #25: foo (line 1, character pos4:7 )
Token #24: := (line 1, character pos8:10 )
Token #25: foo (line 1, character pos11:14 )
Token #12: + (line 1, character pos15:16 )
Token #29: 1 (line 1, character pos17:18 )
Token #4: ; (line 1, character pos19:20 )
// a comment...
/* another multi-line
* java style comment!
*/
foo := 42;
Token #25: foo (line 6, character pos86:89 )
Token #24: := (line 6, character pos90:92 )
Token #29: 42 (line 6, character pos93:95 )
Token #4: ; (line 6, character pos95:96 )
Token #0: (line 7, character pos97:98 )
JLex interfacing
As you were told in lectures, JLex generates a Java class called yylex.
The constructor method for this class takes either a java.io.Reader or java.io.InputStream.
You will note the start of the main program contained in the sample file:
Yylex yy = new Yylex(System.in);
This is how we intend you to instantiate your lexer.
You should not need to do this. However you may want to install it at
home. There are two ways of installing JLex - the easy and the not as easy but not that
hard either way!
The easy way
Download the jlex.jar file into the directory you wish to use
JLex. Simply type:
java -jar jlex.jar your-lex-file
That's it!
The not as easy way
Copy the file Main.java from our JLex directory.
Create a directory called "JLex" somewhere. You'll need the parent of
this directory to be on your Java CLASSPATH. On my own (Linux) based system, I have a
directory ~kevin/javastuff which is on my CLASSPATH. I created a JLex
subdirectory in there. Compile the JLex system by:
javac Main.java
This will create a bunch of Java class files in the JLex directory. Hopefully, without
error.
Here is an ls -l of the Jlex directory on this computer:
-rw-r--r-- 1 kevin cs 297 Mar 10 09:13 CAcceptAnchor.class
-rw-r--r-- 1 kevin cs 641 Mar 10 09:13 CAccept.class
-rw-r--r-- 1 kevin cs 1377 Mar 10 09:13 CAlloc.class
-rw-r--r-- 1 kevin cs 455 Mar 10 09:13 CBunch.class
-rw-r--r-- 1 kevin cs 530 Mar 10 09:13 CDfa.class
-rw-r--r-- 1 kevin cs 479 Mar 10 09:13 CDTrans.class
-rw-r--r-- 1 kevin cs 19731 Mar 10 09:13 CEmit.class
-rw-r--r-- 1 kevin cs 2691 Mar 10 09:13 CError.class
-rw-r--r-- 1 kevin cs 1371 Mar 10 09:13 CInput.class
-rw-r--r-- 1 kevin cs 26171 Mar 10 09:13 CLexGen.class
-rw-r--r-- 1 kevin cs 6092 Mar 10 09:13 CMakeNfa.class
-rw-r--r-- 1 kevin cs 5872 Mar 10 09:13 CMinimize.class
-rw-r--r-- 1 kevin cs 5227 Mar 10 09:13 CNfa2Dfa.class
-rw-r--r-- 1 kevin cs 1031 Mar 10 09:13 CNfa.class
-rw-r--r-- 1 kevin cs 281 Mar 10 09:13 CNfaPair.class
-rw-r--r-- 1 kevin cs 1297 Mar 10 09:13 CSet.class
-rw-r--r-- 1 kevin cs 2465 Mar 10 09:13 CSimplifyNfa.class
-rw-r--r-- 1 kevin cs 2936 Mar 10 09:13 CSpec.class
-rw-r--r-- 1 kevin cs 2677 Mar 10 09:13 CUtility.class
-rw-r--r-- 1 kevin cs 666 Mar 10 09:13 Main.class
-rw-r--r-- 1 kevin cs 203947 Mar 10 09:13 Main.java
-rw-r--r-- 1 kevin cs 605 Mar 10 09:35 Sample.class
-rw-r--r-- 1 kevin cs 4712 Mar 10 09:38 sample.lex
-rw-r--r-- 1 kevin cs 15829 Mar 10 09:33 sample.lex.java
-rw-r--r-- 1 kevin cs 356 Mar 10 09:13 SparseBitSet$1.class
-rw-r--r-- 1 kevin cs 356 Mar 10 09:13 SparseBitSet$2.class
-rw-r--r-- 1 kevin cs 356 Mar 10 09:13 SparseBitSet$3.class
-rw-r--r-- 1 kevin cs 922 Mar 10 09:13 SparseBitSet$4.class
-rw-r--r-- 1 kevin cs 191 Mar 10 09:13 SparseBitSet$BinOp.class
-rw-r--r-- 1 kevin cs 5952 Mar 10 09:13 SparseBitSet.class
-rw-r--r-- 1 kevin cs 1008 Mar 10 09:35 Utility.class
-rw-r--r-- 1 kevin cs 8442 Mar 10 09:35 Yylex.class
-rw-r--r-- 1 kevin cs 816 Mar 10 09:35 Yytoken.class
If you ignored the advice above about obtaining JLex from the Department's JLex
directory and instead forged off on your own and downloaded JLex from the web - you may
now be sitting looking at a bunch of Java warnings (that will recurr each time you use
JLex. Copy our version. Like we said. Life is too short!
You should now have JLex installed, as it were. Make sure your Java CLASSPATH now
points to the parent of the JLex directory. Make sure it doesn't point to
JLex!
