Question Log writing is very slow

[koteswara rao]

Hi,
When i writing log in to text file its was not in the real time. Please Help in this task..
Writing thread

private void WriteEnggLog(string CompleteData)
{
    try
    {
        extraData2 += CompleteData;

        CompleteData = extraData2.Replace(startsWith3, "|");

        string[] packts = CompleteData.Split('|');

        extraData2 = string.Empty;

        for (int rg = 0; rg < packts.Length; rg++)
        {
            if (packts[rg] != "")
            {
                string Data = ( startsWith3 + packts[rg]);

                if (Data.Length == 1588)
                {


                    strHexaLog = FCU_Helper.HextoAscII(Data.Substring(0, 8)).PadRight(10);
                    strEngLog += Littleendinfor2BytesForRev(FCU_Helper.HextoAscII((Data.Substring(0, 8))))+"\t";

                    double xtime = Convert.ToDouble(FCU_Helper.Hex2Dec(Data.Substring(8, 4)));
                    strHexaLog += "0x" + Data.Substring(8, 4).PadRight(10);

                    xtime = xtime * 0.01;
                    strEngLog += OutAddzerostostring(xtime.ToString(),8).ToString()+"\t";

                    for (int i = 1; i < 259; i++)
                    {
                        try
                        {
                            string TextBoxValue = string.Empty;
                            string OutVal = string.Empty;

                            {
                                string[] positionForLog = paramPosForLog[i].Split(',');
                                string val1 = (Data.Substring(Convert.ToInt32(positionForLog[0]), Convert.ToInt32(positionForLog[1])));
                                strHexaLog += "0x" + val1.PadRight(10);

                                TextBoxValue = Expression3(val1, FarmulaparametersForLog[i]);


                                if ((i >= 240))
                                {

                                    OutVal = (Convert.ToDecimal(TextBoxValue).ToString());
                                }
                                else
                                {

                                    if (FlagofIEEEForLog[i] == "IEEENOTOK")
                                    {

                                        if (i < 240)
                                        {



                                            if (TextBoxes3[i] == "GCUAzAchieved" ||
                                                TextBoxes3[i] == "GCUElAchieved" ||
                                                TextBoxes3[i] == "GCUAzCommanded" ||
                                                TextBoxes3[i] == "GCUElCommanded")
                                            {
                                                OutVal = ValidationforSineWaveForGCU2(FCU_Helper.Hex2Dec(val1).ToString());
                                                OutVal = OutAddzerostostring(OutVal, 8);

                                            }
                                            else if (TextBoxes3[i] == "ACUCommandedMotor1" ||
                                                     TextBoxes3[i] == "ACUCommandedMotor2" ||
                                                     TextBoxes3[i] == "ACUCommandedMotor3" ||
                                                     TextBoxes3[i] == "ACUCommandedMotor4" ||
                                                     TextBoxes3[i] == "ACUAchievedMotor1" ||
                                                     TextBoxes3[i] == "ACUAchievedMotor2" ||
                                                     TextBoxes3[i] == "ACUAchievedMotor3" ||
                                                     TextBoxes3[i] == "ACUAchievedMotor4")
                                            {
                                                OutVal = ValidationforSineWaveForACU2(FCU_Helper.Hex2Dec(val1).ToString());
                                                OutVal = OutAddzerostostring(OutVal, 8);
                                            }
                                            else
                                            {

                                                OutVal = Math.Round(Convert.ToDecimal(TextBoxValue), 8).ToString();
                                                OutVal = OutAddzerostostring(OutVal, 8);
                                            }
                                        }
                                        else
                                        {
                                            OutVal = Math.Round(Convert.ToDecimal(TextBoxValue), 8).ToString();
                                            OutVal = OutAddzerostostring(OutVal, 8);
                                        }
                                    }
                                    else
                                    {
                                        // OutVal = Math.Round(FCU_Helper.validateHextoFloat(val1), 4).ToString();
                                        OutVal = Math.Round(FCU_Helper.validateHextoFloat(val1), 8).ToString();
                                        OutVal = OutAddzerostostring(OutVal, 8);
                                    }
                                }


                                strEngLog += OutVal + "\t";
                            }

                        }
                        catch (Exception exG)
                        {
                        }
                    }



                    string hexafpath = FcuMain.FCULogsPath + logfile1 + "/HexaLog";
                    string hexpath = hexafpath + "\\" + logfile1 + "_" + DateTime.Now.ToString("ddMMMyyyy_HH") + "_" + Quniquefile2 + ".txt";

                    using (StreamWriter stream = new FileInfo(hexpath).AppendText())
                    {
                        stream.WriteLine(strHexaLog);
                    } string engfpath = FcuMain.FCULogsPath + logfile1 + "/EngLog";
                    string engpath = engfpath + "\\" + logfile1 + "_" + DateTime.Now.ToString("ddMMMyyyy_HH") + "_" + Quniquefile2 + ".txt";

                    using (StreamWriter stream = new FileInfo(engpath).AppendText())
                    {
                        if (headingCounter == 0)
                        {

                            string Configuration = HeaddingsConfigData + FCU_Health.FCUHealthLog + Environment.NewLine + FCU_IMUconfig.ImuConfigLog + Environment.NewLine + FCU_ConfigVN100ADIS.VN100ADISLog + Environment.NewLine + FCU_ConfigVN100ADIS.MissileConfigLog;
                            stream.WriteLine(Configuration + "\n");

                            string[] headings = TestFile_Eng.Split('$');
                            stream.WriteLine("Header  " + headings[0].Replace(",", " ".PadRight(4)));
                        }
                        stream.Write(strEngLog);
                    }
                    strEngLog = "";

                    headingCounter++;


                }
                else
                {
                    extraData2 = Data;
                }
            }
            strEngLog += Environment.NewLine;
        }


    }
    catch (Exception ex)
    {
    }
}

 

[jmcilhinney]

It's not really for us to test and debug your code for you. You need to do the testing and debugging yourself and then, if you can't work out the solution, tell us where the issue is and we can have a go. Debug your code and see whether it does what you expect. Time sections of your code and see where the time is being taken. Once you have this sort of information, then come back and provide it to us. Also, if you have complex code that doesn't work as expected, you should ALWAYS strip it back to the simplest code possible and then build it up bit by bit until it breaks and then you know which part is the issue. That you're coming to us with code that is that complex and not working shows that you're not really building it the way you should in the first place.

 

[koteswara rao]

Dear jmcilhinney,
My windows application is receiving 796 bytes continuously with 10 mil sec data rate on serial port. we need to display 36 plots and 256 textboxes at a time, displaying and plotting is smoothly running. but when ever i am writing data to text file it was very slow to write in text file. What ever the code i mentioned in previous thread that we need for writing the log.

 

[Skydiver]

Just scanning the code, some of the things that I see as potential slow downs are:

• All the string concatenation to build up a string for output. Use a StringBuilder to do all the appends, and just instantiate the string once right before output.
• You are opening and closing two different streams each time through your loop. Just open the streams outside the loop.
• But you say that the file name changes based on the current time. I ask you do you really need that level of precision for the filenames since your name's precision is down to the hour, but your data is coming in 100 times a second? Does it matter that much if your loop transitions into the next hour while you are in the middle of processing the current log entry?
• Just like above with the filename, it looks like there are some things which will be constant while within the loop. Compute those outside the loop.
• You have multiple branches within the loop. Branches kill the ability of the processor to successfully pipeline commands. Later, it may make sense to have the branches outside, and they invoke different kinds of loops with fixed logic within them, instead of the loop trying to determine which logic to execute at that time.
• Splitting strings causes memory allocations. Memory allocations are slow. Can you just scan through the string and parse them instead?
• It looks like there are some things that you recompute multiple times instead of just computing once. ex. FCU_Helper.HextoAscII(Data.Substring(0, 8)) on lines 23, and then computed again on line 24.
• It looks like there are places you convert a number into a string (and therefore incur memory allocation), only to turn around and convert it back again into a number doing more manipulations and then convert back again to string.
• It looks like there are places where you have a string, convert it into a number, only to turn around and convert it back into a string again.
• Unnecessary calls to ToString().

 

[Skydiver]

Also, external to the code, make sure that you disable your anti-virus software from scanning your log files. Depending on which software you are using, some of them will block your process until the scan is completed. Others will put a lock on the file until the scan is completed preventing you and other applications from accessing the file for that duration.

 

[Skydiver]

And another thing: That try-catch from lines 32-112 can be potentially expensive whenever an exception is thrown because recovering from an exception is very slow. Instead of using Convert(), use the various flavors of TryParse() instead.

 

[koteswara rao]

Skydiver,​

Thank you