Panorama > Code > Program Access to Databases
 
importtext
importtext
TEXT
,
OPTION
,
VALUE

The importtext statement imports text into an existing database.

Parameters

This statement has three parameters:

text – text to import. If you want to import text from a file, use the fileload( function to generate this parameter value.

option – An option for customizing the way the import works. See below for descriptions of each option. The option can be specified in either upper or lower case.

value – The new value for this option.

Description

This statement imports text into the current database. The text should be organized into fields separated either by commas (often called comma separated text, or CSV), or by tabs. In its simplest form you simply supply the data to be imported:

importtext fileload("Update 2014-08-12.csv")

Advanced Note: The procedure above will automatically handle the text properly if it is encoded using UTF-8 (the default for OS X applications), Mac OS Roman (the default for OS 9 applications), or UTF-16. If you need to import text that has been saved with a different encoding you must explicitly convert the data to text using the binarytotext( function. This example will import data that has been encoded in CP-1252, a format commonly used on Windows:

importtext binarytotext(fileload("Update 2014-08-12.csv"),"WindowsCP1252")

If the imported text contains more columns than the database, the importtext statement normally adds the extra fields needed to the database automatically. This can be disabled, see the addfields option below. The new fields are named alphabetically: A, B, C, …, AA, AB, etc. This makes it very easy to create a new database with imported data. This example will create a new database and fill it with seismic data – automatically creating whatever fields are necessary.

newdatabase
importtext fileload("SeismicData.csv")

If different rows in the imported data contain different numbers of columns, the number of new fields generated will be based on the imported row that has the most columns. So if most of the imported text contains 6 columns, but row 749 contains 9 columns, the database will wind up with 9 columns (unless it had more than 9 to start with).

IMPORT OPTIONS

In addition to the text to import, you can also specify multiple option/value pairs in a single importtext statement, like this:

importtext text,option1,value1,option2,value2,option3,value3

The possible options are separator, quote, existingdata, addfields and rearrange. Each of these options is described in detail below.

SEPARATOR

This option is used to explicitly specify the column separator character, for example comma, tab, or semicolon.

importtext sometext,"Separator","," ☞ comma separated
importtext sometext,"Separator",tab() ☞ tab separated
importtext sometext,"Separator",";" ☞ semicolon separated

If the separator parameter is omitted, Panorama will automatically choose the separator to use. It does this by scanning the first line of the text – if finds one or more tabs, it assumes the text is tab separated. If no tabs are found, it normally assume the text is comma separated. However, if the decimal point is configured to comma instead of period, it will assume the text is semicolon separated. (The decimal point character is controlled by the Language & Region panel in the System Preferences window.)

QUOTE

When importing comma or semicolon separated text, the importtext statement normally ignores commas or semicolons inside quoted text. This allows commas and semicolons to appear in the data itself, like this:

"Washington, George",Virginia
"Adams, John",Masachusetts
"Jefferson, Thomas",Virginia

Quoted text can also contain quotes, like this:

"Washington said ""I cannot tell a lie""",false

This option allows you to change the quote character, or you can specify "" to disable this special quote processing altogether, like this:

importtext Presidents,"Quote",""

Note: Special quote processing is disabled when importing tab delimited text (or any separator other than comma or semicolon), you can use this option to enable it.

EXISTINGDATA

The importtext statement normally appends the imported data to the end of the current database. You can use this option to tell it to replace the data instead.

importtext Presidents,"ExistingData","Replace"

Be careful! This will erase all of the existing data in the database! (Note: The replace option is not available for shared multi-user databases, attempting to use this option will cause an error.)

If you want to append the imported data, use append instead of replace (or just leave this option off).

ADDFIELDS

As described above, the importtext statement normally adds extra fields as necessary to hold all of the imported data columns. If you don’t want this to happen, set this option to false (or "NO"). For example, if you have a carefully crafted inventory database, you won’t want a text file with stray fields to add extra fields on the end.

importtext fileload("Inventory Update.csv"),"AddFields","NO"

Note: If adding fields is disabled, Panorama will generate an error if the text contains extra columns that won’t fit in the existing fields. However, it will still import the data that does fit.

Note: If you are importing into a shared multi-user database (connected to a server), you must disable this option, since fields can’t be added to a shared database (except through the “new generation” process.) Because of this restriction, when using a shared database this option defaults to NO if the option is omitted.

FIELDTYPE

Any extra fields added by this statement are usually text fields. However, if you are absolutely sure that the data you are importing is numberic, you can use the fieldtype option to specify that any new fields created are either integers or floating point. This saves you the additional task of converting the field type after the import. The available options are either "INTEGER" or "FLOAT", as shown in this example:

importtext fileload("Monthly Sales.csv"),"FieldType","Float"

Remember, this only applies to any new fields that are added at the end of the database, it does not affect any existing fields. You can only specify one field type, so you can’t mix integer, float and text fields. If there is any doubt of the data type of the imported information the safest approach is to import the data as text and then convert it to numeric afterwards.

REARRANGE

The importtext statement normally imports data in left to right order – the leftmost column of the imported text is imported into the first database field, the next column is imported into the second field, etc. The rearrange option allows this order to be changed, or can even be used to combine, ignore, or otherwise manipulate the imported data as it is brought into the database. (Legacy note: The rearrange option performs the same function as the importusing statement in previous versions of Panorama.)

Using the rearrange option, you specify a formula that is used to process the import data. This formula is “inserted” into the middle of the import process. The formula must be designed to take in a line of text, then transform it into a different line of text. The importtext statement uses this formula to transform each raw line of the import data into a new, manipulated line. This new manipulated line is then imported into the database instead of the original line.

Panorama has a two special functions that allow the import translation formula to access the line that has been read from the disk: import( and importcell(. The import( function returns the entire line that has been imported. The importcell( function has one parameter that specifies the number of the cell you want, for example importcell(1) or importcell(14). (Note that importcell( uses 0 for the leftmost field, not 1).

Suppose you have a text file named Sam’s Contacts that contains data like this:

Smith,John,World Widgets,124 W. Olive St,San Jose,CA,95134
Lee,Susan,Industrial Metals,2347 N. Riverside,Cambridge,MA,02139
Marklee,Lance,Zipper Technologies,687 E. Dorothy Lane,Bothell,WA,98011
Anders,Fred,Acme Fireworks,5672 Lakewood Drive,Salinas,CA,93908

You want to import this data into a database that contains these 8 fields in this order:

Company, First Name, Last Name, Title, Address, City, State, Zip

The data needs to be rearranged as it is being imported, otherwise the last name will wind up in the Company field, the street address will end up in the Title field, etc. Here’s a procedure that will append the data in Sam’s Contacts into the current database. (Note: The separator in the rearrange formula must always be a tab, even if you are importing comma delimited text.)

importtext fileload("Sam's Contacts"),
    "REARRANGE",
        importcell(2)+tab()+
        importcell(1)+tab()+
        importcell(0)+tab()+
        tab()+
        importcell(3)+tab()+
        importcell(4)+tab()+
        importcell(5)+tab()+
        importcell(6)

The formula rearranges the incoming data so that third column in the input text goes into the first field, the 2nd column goes into the 2nd field, the first column goes into the 3rd field, the 4th field is empty, the 4th column goes into the 5th field, the 5th column goes into the 6th field, the 6th goes into the 7th field and the 7th column goes into the 8th field.

In the example above, each column in the input corresponds with one field in the final database. However, you could split up a column into multiple fields, or combine multiple columns in the input text into a single field in the final database. For example, suppose you wanted to import Sam’s Contacts into a database with the fields:

Name, Address, City, State, Zip.

Here’s the procedure you would need.

importtext fileload("Sam's Contacts"),
    "REARRANGE",
        importcell(1)+" "+importcell(0)+tab()+
        importcell(3)+tab()+
        importcell(4)+tab()+
        importcell(5)+tab()+
        importcell(6)

This example simply concatenates the first and last names with a space, but you can use any function you want, including the ?(, sandwich(, upper(, lower(, even lookup( functions. Here’s a slightly revised example:

importtext fileload("Sam's Contacts"),
    "REARRANGE",
        upper(importcell(0))+", "+importcell(1)+tab()+
        importcell(3)+tab()+
        importcell(4)+tab()+
        importcell(5)+tab()+
        importcell(6)

With this procedure, the Name field will wind up with names looking like this:

SMITH, John
LEE, Susan
MARKLEE, Lance
ANDERS, Fred

The import( function returns the raw imported line. For example, suppose you wanted all of the imported data to be converted to upper case.

importtext somedata,"REARRANGE",upper(import())

Advanced Note: Since the import( function returns the raw text, no special quote processing will be done. So the example above will not work properly if your imported data contains comma separated data with quotes around any of the fields. You can fix this with the csvtotsv( function, which converts comma separated text to tab separated text, and does the special quote processing.

importtext somedata,"REARRANGE",csvtotsv(upper(import()))

The import( function is especially useful for importing text with no delimiters at all, for example text that is formatted into fixed width columns (with each column padded by space characters) like this. The first column is 16 characters, the second is 10, and the third is whatever is left.

100 Meter Dash  Gold      Edwards, Mark
100 Meter Dash  Silver    Franklin, Lee
100 Meter Dash  Bronze    Gonzales, Esteban
Pole Vault      Gold      George, Michael
Pole Vault      Silver    Rafferty, Robert
Pole Vault      Silver    Delgado, Marquise

This procedure will import this fixed width data into a database containing the fields Event, Medal, and Name.

importtext results,"REARRANGE",
    strip(import()[1;16])+tab()+
    strip(import()[17;10])+tab()+
    strip(import()[27,-1])

This procedure uses text funnels to extract the fixed width data, see Text Funnels for more information.

See Also

History

VersionStatusNotes
10.2UpdateThis statement now uses semicolon as the default CSV separator if the system is configured to use comma as the numeric decimal point. Also, if the database is shared (multi-user), you cannot use the ADDFIELDS or REPLACE option with this statement, and ADDFIELDS defaults to NO for shared databases.
10.0NewNew in this version, but includes features that were part of the *OpenFile* and *ImportUsing* statements in Panorama 6 and earlier versions.