Import tool

This is a tool which can be used to easy importing data into a equanda application.
The program scans a directory for import files to appear (very configurable) and imports them using ImportServlet.
There is special handling to allow errors to be caught and corrected.
Alternatively, this program can also execute scripts which can be used to build the import file at runtime
(thus allowing a direct link with JDBC data sources etc).
This option is typically used to synchronize with another application
(one-way synchronization, use the Export tool for the other direction).

Usage

The import tool can either scan directories for import files to appear,
or can run scripts which produce the import file (probably by reading data from a proprietary system).

The file format for importing is ImportFileFormat.

General configuration

The tool (class org.equanda.tool.ymport.ImportClient) accepts one optional parameter,
the name of an .ini file which is used for the configuration.
If no parameter is given, the program tries to open "import.ini".

The .ini file has the following format

    [general]
    scanPeriod=30
    outputExtension=err
    window=display
    search=local
    importURL=http://192.168.0.252:8080/admin/import/import

    [mail]
    server=mail.paragon-software.ro
    port=110
    user=test@paragon-software.ro
    password=testtest
    directory=/data/
    accept=florin@paragon-software.ro;kunglao_99@yahoo.com
    connectionTimeout=30
    socketTimeout=30

    [scan]
    dir1=c:\import
    input1=*.txt
    charset1=ISO-8859-1
    dir2=g:\import
    input2=*.import
    script2=import.isc

    [script]
    debug=false
    script1=weight.isc
    script2=article.isc

	[database]
	name1=vlees
	driver1=sun.jdbc.odbc.JdbcOdbcDriver
	location1=jdbc:odbc:CodeBase Datasource;database=C:\dbf\
	user1=
	password1=

This has the following meaning

general section

• scanPeriod : time in seconds between attempts to pick up files to import or attempt to run the import scripts. Defaults to 60s.

• outputExtension : extension for output (error) files. The extension of the input file is replaced, or the extension is added if the input file had no extension.

• window : indicates whether a window needs to be displayed or not (values "display" or "hidden").

• search : indicates whether the program should only scan in the scan directories, or also in subsidrectories. Possible values are: "deep" and "local".

mail section

This allows mails to be read which can then be imported. The attachments will be placed in the chosen location for further processing (scanning). Only mails from the list of allowed senders are accepted (slight security measure as anyboy could send mails to the designated mailbox.
{warning}

Accepting data to be imported from mails is always a security risc as sender addresses are very easy to forge. Once somebody knows the mail address and sender address top use, they can inject and overwrite anything in the database.
{warning}

• server : mail server to get mails from

• user : user to get mails for

• password : password for the user

• accept : senders for which the emails are accepted

• directory : location where the attachments need to be saved

• connectionTimeout, socketTimeout : timeouts for connection, in seconds, default is 30

scan section

• This can contain the directories which have to be scanned for input files (scanX with X a number) and a regular expression (inputX) indicating the files which need to be imported (default is *), for the directory with the same number

• It may also contain the charset to be used for the files from the matching directory. When charset is not set, the system default is used.

• For each level, an import script file can be specified. The "build(string filename)" function is called in this script to allow conversion from other types of files.

script section

• debug : can be "true" or "false". Normally the data which is built is immediately sent to the import servlet. However, when debug is "true", then the data is not imported but simply written to an output file in the current directory.

• scriptX (with X a number) : name of a script file which builds the data to be imported.

• database section

• As with script and scan, the names need to end with a number to indicate what belongs together. Counting starts at one and should be continuous. The database section is only relevant for import scripts.

• name : name for the database, as used in "Dbset(var,name)"

• driver : JDBC driver class

• location : database url for connection with the database

• user, password : username and password for the connection

config section

• In the config section extra kay/value pairs can be put which can be retrieved in the scripts using "config(key)".

Program functioning

The import itself starts by computing the name of the error file
(the extension (everything after the last dot, if any) is replaced by "outputExtension").
If the error file exists, the import file is skipped.
It then tries to open the import file in read/write mode.
If that fails, the file is skipped (it will be picked up again in the next scan).
When the file is opened, the contents is sent to the import servlet.
If the import was successful, then the input file is deleted.
If the import failed, the return message from the servlet is written in the error file.

Script handling

The program accepts FScript scripts to build the data to be imported.
The script should have the following structure :

func build()
	# this functions is called to builds the import file
	# You have the following commands for this
	# table(selectorName, commaSeparatedFields)
	# field(data) : add a field value
	# line([string]) : next line or add line with given text
	# quote(string) : quote a string using single quotes, doubles the single quotes in the string
	# import() : import the current data, returns whther the import was succesful
	#       (allows forcing import halfway through the script)
	# if references need storing,
	# use the "list" variable (of type "List")
endfunc

func onSuccess()
	# this function is called when the import was succesful
	# can be used to mark te data as imported
	# (again, referencing the contents of "list")
endfunc

func onError(string error)
	# this function is called if the import failed
	# the "list" data is still available
endfunc

The FScript extensions from the util module are available.