Analytics » Upload Tool » Uploading Data from CSV Files - GUI Mode

Uploading Data from CSV Files - GUI Mode

Upload Tool can be invoked using Command line mode (console mode) or with a user interface (GUI). GUI mode can be useful if you are not much familiar with the command line interface. In this section, we will discuss how to set up this tool for batch upload of CSV data with a user interface.

 Note:

The Upload tool will be useful, when you need to upload more than 500,000 rows or whose file size is greater than 50 MB (these import size limits are applicable when you use the Virtutem Analytics web user interface). This tool splits the data in the CSV file into batches and uploads it to Virtutem Analytics, making it much more resilient to intermittent Internet connectivity outages or slow connections

   Uploading Data

Steps for running the CSV upload tool in GUI mode.

    • Step 1:  Ensure that you have already created the database in  Virtutem Analytics  account into which the data is to be uploaded.

    • Step 2:  Configure the proxy settings and the required Import settings in the  common_params.conf  file. Read more about these settings from Proxy Settings and  Import Settings sections. 

    • Step 3:  Invoke the  CSVUploadGUI.sh  (for Linux/Mac)  or CSVUploadGUI.bat  (for Windows)   file under  <Tools_Home>bin  directory. The  CSV Upload Utility  dialog will open. 

    • Step 4:  Provide the required inputs in the GUI as shown below.

  • Step 5:  Click the  Upload  button.

  • All your data from the CSV file will get uploaded to the specified table and in Virtutem Analytics database.

The following table contains the parameters that are to be specified in the GUI above.

Parameter Name

Description

File

Specify the complete path of the CSV file from which data is to be uploaded.

Database Name

Specify the name of the database in the Virtutem Analytics account into which the CSV file's data is to be uploaded. Ensure that the database is already available in Virtutem Analytics. In case the database does not exist, contact Virtutem to create a blank database with the same name in your Virtutem Analytics account. 

Table Name

Specify the name of the database table into which the CSV file's data is to be uploaded. Ensure that the specified table with similar column structure is already created in the Virtutem Analytics Reporting Database.

In case the table does not exist, then you can set the Upload Tool to create the table and then upload the data into this table using the  ZOHO_CREATE_TABLE  parameter available in  common_params.conf  file. Refer to the  Configure Import Settings  section to know more about the ZOHO_CREATE_TABLE parameter.

User Name

Specify your Virtutem Analytics account login name.

Note:  If you have registered in Virtutem Analytics using other login such as Google Apps, Gmail, Facebook & Yahoo, then you cannot use these credentials to upload data using Upload Tool. In this case you need to generate a Virtutem Username and Password to upload data. To know how to generate Virtutem User credential, click here. 

Password

Specify the Virtutem Analytics account password.

Import Type

Choose one of the following three options as the  Import Type


·   Add at the end  - The data in the CSV file will be added (appended) at the end of the table.

·   Delete existing and add  - All existing rows in the table will be deleted and then the data in the CSV file will be added.

·   Add, replace if already exists - Existing rows that are modified will be updated and new rows will be appended at the end of the table. (In this case ensure you provide the appropriate input for the field  Columns to match  described below).

Columns to match

This is applicable only when the  Import type  is set to  Add, replace if already exists . Specify column ( or a combination of columns) whose value will uniquely identify each record in that table. If the record already exists in the table then it will be replaced with the new values in the CSV file being uploaded.  Otherwise the data will be added at the end of the table.

 Note:
 The last uploaded details will be logged under  <Tool_Home>/logs/csvupload_serverout.txt  file.

Virtutem Analytics Upload Tool provides more advanced settings that can be configured while uploading data from CSV files. You can learn about these import settings in the following section.

 Configure Import Settings

The configuration file  common_params.conf  in the      /conf  directory provides parameters that Upload Tool uses while uploading data. These settings help Virtutem Analytics properly understand the data being imported.


The following table lists the import parameters that can be configured in the  common_params.conf  file for uploading data from CSV files using the GUI mode. 


Note : You will not find all the below given parameters present in the configuration file (except mandatory ones). In case you do not find them, add them into the file and provide the required value as specified below. 

Parameter

Possible Values

Description

ZOHO_CREATE_TABLE

·   True

·   False

Optional
This parameter specifies whether to create the table or throw error, if the specified table does not exist in the database.

·   True  - New table will be created in the specified database with the said name and then the data will be imported. In case the table already exists in the database, then the data will be imported into it. 

·   False  - The data will be imported into the specified table. In case the table does not exist in the specified database then it throws an error.

Default value is  false.

LINES_TO_SEND

 

·   True

·   False

Optional 
Upload Tool splits the CSV file into batches for uploading. This parameter specifies the number of row for each batch.
Default value is  5000 .

ZOHO_FIRST_ROW_HEADER

 

Optional
This parameter specifies whether the first row of the CSV file is the column header or not. 

·   True  - First row contains the column header names.

·   False  - First row does not contain the column headers. In this case it is mandatory to specify the column header names for the parameter ZOHO_COLUMN_HEADER_NAMES (described next).

Default is  true .

ZOHO_COLUMN_HEADER_NAMES

Column Names with comma separator

This parameter is mandatory when the  ZOHO_FIRST_ROW_HEADER  is set to  false
Specify the column headers/names in the same order as the data exists in the CSV file. The column names provided here will be considered as the header of the CSV file.

ZOHO_DATE_FORMAT

Format of the date.
E.g., 
ZOHO_DATE_FORMAT="dd-MMM-yyyy

Note: 

The above example indicates a date in the format 10-Nov-2013. Ensure you provide the appropriate notation to match your date format.

Optional 

In case you are using a custom date format, in the data being uploaded, other than the default date format in Virtutem Analytics, then you can use this parameter to specify the date format. Refer to the following link: How to construct date format string. 
 

ZOHO_AUTO_IDENTIFY

·   True

·   False

This parameter specifies whether the tool should automatically identify the delimiter or not.

·   True  -The tool will automatically identify the delimiter (ZOHO_DELIMITER parameter) and text qualifier (ZOHO_QUOTED parameter) in the CSV file that is being imported. By default this will be set to  true

·   False  - You need to specify the delimiter and text qualifier explicitly. 

In case this parameter is set to false, then you need to specify the delimiter and text qualifier using the parameters in the  table below.

ZOHO_SKIPTOP

 

Optional
This parameter specifies the number of top rows that are to be skipped in the CSV file being imported.

ZOHO_THOUSAND_SEPARATOR

·   0

·   1

·   2

·   3

Optional

This parameter specifies the thousand separator for the numeric data.
The following is the list of parameter values and the corresponding thousand separator.

·   0 - COMMA

·   1 - DOT

·   2 - SPACE

·   3 - SINGLE QUOTE

Default is  0 .

ZOHO_DECIMAL_SEPARATOR

·   0

·   1

Optional

This parameter specifies the decimal separator for the numeric data.

The following is the list of parameter values and the corresponding decimal separator.

·   0 - DOT

·   1 - COMMA

Default value is 0.

ZOHO_FILE_ENCODING

·   UTF-8

·   UTF-16

·   GB18030

·   Shift_JIS

Optional 
This parameter specifies the File Encoding Format of the input file.

Default value is  UTF-8 .

ZOHO_REMOVE_BOM

·   True

·   False

Optional
If you are uploading a CSV file which contains Byte Order Mark (BOM), then you can use this parameter to identify and remove BOM Characters.

Set this to  true , the tool will identify the BOM and remove the BOM from the file. You can also specify the BOM type (UTF-16LE,UTF-16BE), if you know the type.

ZOHO_CANOVERRIDE

·   True

·   False

Optional 

In case another import is triggered for the same table when this import is in process, then you can use this parameter to specify whether this import  process can override the other import running. 

·   True - The tool will stop the current import process and overwrite the table with new import.

·   False - The tool will not disturb the current import process and will cancel the new import request.

Default value is  false.

ZOHO_ON_IMPORT_ERROR

·   ABORT

·   SKIPROW

·   SETCOLUMNEMPTY

This parameter controls the action to be taken in case there is an error during import.

·   ABORT  - In case of any error, the import will be aborted.

·   SKIPROW  - In case of any error,the specific row(s) in which error occurred will be skipped and then continue importing the rest of the data.

·   SETCOLUMNEMPTY  - In case of any error, the value of the specific cell(s) in which error occurred will be set to empty and then continue importing the rest of the data. 

Default value is  SETCOLUMNEMPTY

 Setting Delimiter


The following table lists the parameters that need to be configured to set the delimiter when  ZOHO_AUTO_IDENTIFY   is set to   false.

Parameter

Possible Values

Description

ZOHO_COMMENTCHAR

 

Comment Character.

The CSV row will be skipped, If the comment character is found at the beginning of the row.
Example: ZOHO_COMMENTCHAR="$

The above will skip the rows starting with $ character while importing. 

ZOHO_DELIMITER

·   0

·   1

·   2

·   3

Specifies the delimiter which separates the values in the file. The following is the list of parameter values and the corresponding delimiter:

·   0 - COMMA

·   1 - TAB

·   2 - SEMICOLON

·   3 - SPACE

ZOHO_QUOTED

·   0

·   1

·   2

Specifies the Text Qualifier which surrounds string values in the CSV file, if available. The following is the list of parameter values and the corresponding text qualifier:

·   0 - None

·   1 - SINGLE QUOTE

·   2 - DOUBLE QUOTE

The following screenshot illustrates the  common_params.conf  file showing some of the import parameters:

 Viewing the Data Online

To view the data that you have uploaded

  • Login to  http://analytics.virtutem.com.
  • Click on the corresponding Database name under  My Databases.
  • Click on the corresponding table at the Left to open the table and view the uploaded data.

 FAQ and Troubleshooting

Frequently Asked Questions  

Troubleshooting Tips