Compress Dynamic Files Using 7-Zip in SSIS

Abstract

This blog post is in response to a request/comment from Thato in this previous article. Hence, I have taken much of the logic used in the aforementioned article and changed it from static to dynamic file compression. Thus, I am still using SQL Server Integration Services‘s Execute Process Task to compress dynamic files into a .rar format using 7-Zip.

Requirements

Article

Let’s begin by launching Microsoft Visual Studio and create a new Integration Services Project which is located under Business Intelligence Projects category. (If you are using SQL Server 2012 or later, you will have to launch SQL Server Data Tools – SSDT).

After you have assigned a project name, lets add some variables that will be used to store the dynamic content. I setup the variables as follows:

  • var_FileName (is of type String and stores file name and extension)
  • var_WorkingDIR (is of type String and stores path for working directory)
  • var_NewFileName (is of type String and stores a derived archive file name)
  • var_Exec (is of type String and stores the path to the executable 7-Zip file)

Now let’s click and drag the ForEach Loop Container (FLC) into Control Flow pane from toolbox (SSIS Toolbox in SSDT).

Right click FLC and click on “Edit”. Under the Collection tab, ensure that Enumerator is Foreach File Enumerator. Setup Directory as expression and map it to the var_WorkingDIR variable. Lastly, make sure that Name and extension is checked under the Retrieve file name group box. In summary, the Collection tab should be configured as below:

Still on FLC, under the Variable Mappings tab map variable var_FileName to the first index 0 as shown below:

The next step is to drag a Script Task (ST) into the FLC. The ST will be used to initialize variable var_NewFileName. Under the Script tab of the ST ensure that you have configured the read/write variables as shown below:

Still on ST, click Edit Script. Reference the System namespace as shown below:

The rest of the code in the Main class should be as follows:

The final step is to drag and drop the Execute Process Task (EPT) into Control Flow pane from toolbox (SSIS Toolbox in SSDT).

Right click EPT and click on “Edit”

Under the General Tab, you can assign relevant Name & Description. In this case I have decided to name the task as “EPT – Compress Dynamic Files” whilst Description remains unchanged.

In the Process Tab, the EPT properties must be set as follows:

  • RequiredFullFileName : True
  • FailTaskIfReturnCodeIsNotSuccessValue : True
  • SuccessValue : 0
  • Timeout : 0
  • WindowStyle : Hidden

Still on the EPT, we now need to configure expressions. Thus the expression tab should look as follows:

 

Basically some of the noticeable things we have done above is to hide the exe window that appears when the file is being compressed. The Executable and WorkingDirectory properties are set via variables. The Arguments property is also set using variables and we have further hard coded the extension of the compressed file to be that of type rar. Luckily 7-Zip supports various file compression types, so you can change the rar extension to an extension of your choice. Of course depending on the extension you have provided the size of the new compressed file will vary.

PS: You can download a copy of this package here.

It’s that simple.

Till next time folks, cheers.

Sifiso.

6 Comments

  1. Thato says:

    Hi Sifiso,

    Thanks for the post, it works well in the base folder however it does not process the files that are in subfolders even though I ticked traverse subfolders on the FLC.

    my folder structure is as follows

    C:\Thato\

    File1.txt
    File2.txt
    File3.txt

    C:\Thato\BKP\
    File1.txt
    File2.txt

    C:\Thato\BKP\Prev\
    File1.txt

    currently the package only pick up the files in C:\Thato\ it does pick up subfolders but does not action the files in there

  2. Thato says:

    Hi Sifiso,

    I got it, it was working fine all the time. the problem was that because the files had same names it would replace the existing one in the base folder. So I added a time stamp to ensure that no file is replaced and it’s working fine.

    Thanks a mil.

  3. Thato says:

    Hi Sifiso,

    Just for my understanding, in this instance what does the code in the script task do?

    • Sifiso says:

      i am using the script task to get the new compressed file name based on the original file name. the tricky part is that whenever we extract the compress file name it should not have a file extension because that is further hard coded to be .rar. So the first thing I do is to get an index of the dot (which will indicate the beginning of a file extension). I then do a substring to derive a new compressed file name from a starting point of zero till I reach the beginning of a dot.

  4. Rosemary says:

    Thanks. It works beautifully.

Leave a comment

%d bloggers like this: