Memory Stream Reading Chunks

[madaxe2020]

Is this even possible, I want to construct a memory stream where the first byte defines how the stream should be consumed, the next 4 bytes defines how many bytes the next chunk has and so on

thanks

madaxe

 

[jmcilhinney]

Of course it's possible. You could use a MemoryStream directly but I'd suggest that your best bet would be to use a BinaryWriter to write the data to the stream and then a BinaryReader to read it back.

 

[madaxe2020]

I can send a single byte containing the OpCode and display that on the server in the console great.
So I tried to add 4 bytes to contain the size of the next chunk I could not find an easy way of doing this

I can determine how many bites my chunk is going to have for example if its an image that requires 255 bytes then I really only need 1 byte to store 255 leaving the remaining 3 bytes empty.

byte[] chunksize = { (byte)255,,, };

As far as i know you cant define empty bytes? And having empty bytes maybe wasteful.

So I thought about changing my model and adding one more byte to define the number of bytes required to store the chunk size its self

[OpCode 1 byte] [Chunk Size Storage 1 byte 0-255] [ Chunk Size X Bytes up to 255 bytes] [ Chunk X bytes image ect ] etc

I can repeat this pattern for every chunk I want to put in my stream.

i tried storing and retreiving the following

opcode 1 1 byte
message size 12 1 byte
message Hello World. 12 bytes

when i get the message on the server part of the message is cut off and i dont know why.

Thanks

Madaxe

using Helpers;
using Microsoft.Extensions.Hosting;

using System.Net;
using System.Net.Sockets;

namespace MemoryStreamApp
{
    public class PacketReaderServer : IHostedService
    {
        public async Task StartAsync(CancellationToken cancellationToken)
        {
            TcpListener _tcpListener = new TcpListener(IPAddress.Parse("127.0.0.1"), 7891);
            _tcpListener.Start();

            PacketReader _packetReader;

            while (!cancellationToken.IsCancellationRequested)
            {
                NetworkStream networkStream = _tcpListener.AcceptTcpClient().GetStream();
                _packetReader = new PacketReader(networkStream);

                int opCode = _packetReader.ReadOpCode();

                Console.WriteLine(string.Format("{0} : Opcode Found With Value {1}", DateTime.Now, opCode));

                int nextchunksize = _packetReader.ReadChunkSize();

                Console.WriteLine(string.Format("{0} : Chunk Found With Value {1}", DateTime.Now, nextchunksize));

                string message = _packetReader.ReadMessage(nextchunksize);

                Console.WriteLine(string.Format("{0} : Message Found With Value {1}", DateTime.Now, message));

                await Task.Delay(1000);
            }
        }

        public Task StopAsync(CancellationToken cancellationToken)
        {
            return Task.CompletedTask;
        }
    }
}

public void ConnectToServer(int opCode=0)
        {
            if (!_client.Connected)
            {
                _client.Connect("127.0.0.1", 7891);
                _packetReader = new PacketReader(_client.GetStream());

                _packetWriter.WriteOpCode((byte)opCode);
                _packetWriter.WriteMessage("Hello World.");
                _client.Client.Send(_packetWriter.GetPacketBytes());
            }
        }

using System.Text;

namespace Helpers
{
    public class PacketWriter : BinaryWriter
    {
        public MemoryStream MemoryStream;

        public PacketWriter()
        {
            MemoryStream = new MemoryStream();
        }
        public void WriteOpCode(byte opCode)
        {
            MemoryStream.WriteByte(opCode);
        }

        public void WriteNextChunckSize(byte chunkSize)
        {
            MemoryStream.WriteByte(chunkSize);
        }

        public void WriteMessage(string message)
        {
            int msgLength = message.Length;
            WriteNextChunckSize((byte)msgLength);
            MemoryStream.Write(BitConverter.GetBytes(msgLength));
            MemoryStream.Write(Encoding.ASCII.GetBytes(message));
        }
        public byte[] GetPacketBytes()
        {
            return MemoryStream.ToArray();
        }
    }
}

using System.Net.Sockets;
using System.Text;

namespace Helpers
{
    public class PacketReader : BinaryReader
    {
        private NetworkStream _memoryStream;
        public PacketReader(NetworkStream memoryStream) : base(memoryStream)
        {
            _memoryStream = memoryStream;
        }

        public int ReadOpCode()
        {
            return _memoryStream.ReadByte();
        }

        public string ReadMessage(int chunkSize)
        {
            byte[] messageBuffer;
            messageBuffer = new byte[chunkSize];
            _memoryStream.Read(messageBuffer, 0, chunkSize);

            var message = Encoding.ASCII.GetString(messageBuffer);

            return message;
        }
        public int ReadChunkSize()
        {
            return _memoryStream.ReadByte();
        }
    }
}

 

[jmcilhinney]

Four bytes is 32 bits. An Int32 is a 32-bit number. The BinaryWriter.Write method accepts, amongst other types, an Int32. Guess what the BinaryReader.ReadInt32 metgod does.

 

[madaxe2020]

Four bytes is 32 bits. An Int32 is a 32-bit number. The BinaryWriter.Write method accepts, amongst other types, an Int32. Guess what the BinaryReader.ReadInt32 metgod does.

manages the empty bytes?

i tried this and it seems to work but it closes my memory stream, is this because the binary writer is going to garbage collection due to the using statement?

thanks
madaxe

public void WriteNextChunckSize(int chunkSize)
        {
            using (BinaryWriter writer = new BinaryWriter(MemoryStream))
            {
                writer.Write((Int32)chunkSize);
            }
        }

so i answered my own question and it is because of the using statement so i creates a BinaryWriter property within the constructor and it works thanks

public PacketWriter()
        {
            MemoryStream = new MemoryStream();
            BinaryWriter = new BinaryWriter(MemoryStream);
        }
        public void WriteNextChunckSize(int chunkSize)
        {
            BinaryWriter.Write((Int32)chunkSize);
        }

 

[Skydiver]

No. Read the documentation. BinaryWriter.WriteInt32() will write out 4 bytes in little Endian format that represents the value of an int32 of your machine architecture. BinaryReader.ReadInt32 will read in those 4 bytes in little Endian format and give you back the original value appropriate for your machine architecture.

So if your int was 65, then what would be written out into the stream would be 65 0 0 0 (decimal) or x41 x00 x00 x00 (hexadecimal).

 

[madaxe2020]

No. Read the documentation. BinaryWriter.WriteInt32() will write out 4 bytes in little Endian format that represents the value of an int32 of your machine architecture. BinaryReader.ReadInt32 will read in those 4 bytes in little Endian format and give you back the original value appropriate for your machine architecture.

So if your int was 65, then what would be written out into the stream would be 65 0 0 0 (decimal) or x41 x00 x00 x00 (hexadecimal).

Thanks for the help I'm an engineer not a computer major, so I'm learning a lot.

so I'm now able to write the int32 and read it
so you can see from the output I'm getting the correct Opcode and Chunk size but the message is being cut of any ideas

thanks

madaxe

10/09/2022 09:57:04 : Opcode Found With Value 0
10/09/2022 09:57:07 : Chunk Found With Value 12
10/09/2022 09:57:07 : Message Found With Value Hello W


the problem is with getting the message size, if i add 3 to the msgLength then i get more of the message

public void WriteMessage(string message)
        {
            int msgLength = message.Length+3;
            WriteNextChunckSize(msgLength);
            MemoryStream.Write(BitConverter.GetBytes(msgLength));
            MemoryStream.Write(Encoding.ASCII.GetBytes(message));
        }

public void WriteNextChunckSize(int chunkSize)
{
      BinaryWriter.Write((Int32)chunkSize);
}

public void WriteMessage(string message)
{
      int msgLength = message.Length;
      WriteNextChunckSize(msgLength);
      MemoryStream.Write(BitConverter.GetBytes(msgLength));
      MemoryStream.Write(Encoding.ASCII.GetBytes(message));
}

public string ReadMessage(int chunkSize)
{
     byte[] messageBuffer;
     messageBuffer = new byte[chunkSize];
     _memoryStream.Read(messageBuffer, 0, chunkSize);

     var message = Encoding.ASCII.GetString(messageBuffer);

     return message;
}
public int ReadChunkSize()
{
     byte[] messageBuffer;
     messageBuffer = new byte[4];
     _memoryStream.Read(messageBuffer, 0, 3);
     return BitConverter.ToInt32(messageBuffer,0);
}

 

[madaxe2020]

Hello World byte array
72
101
108
108
111
32
87
111
114
108
100
46

when I try to read the message it looks like its reading the complete memory stream from the start, I can see the opcode (0) 1 byte, chunk size (12 0 0 0) 4 bytes and then the start of the message, but i can see its cut off but i dont know why.

0

12
0
0
0

72
101
108
108
111
32
87

 

[madaxe2020]

changing my Encoding to UTF8 and changing the number of bytes I was reading from 3 to 4 I keep thinking its zero base.

i also found UTF8 worked better than ASCII

_memoryStream.Read(readChunkSize, 0, 4);

thanks

madaxe

public void WriteMessage(string message)
{
    WriteNextChunckSize(message.Length);
    byte[] byteArray = Encoding.UTF8.GetBytes(message);
    BinaryWriter.Write(byteArray);
}

 

[Skydiver]

Why did you ignore the advice to use the BinaryWriter and BinaryReader? Is there a particular reason why you want to write and read from the memory stream directly and use the BitConverter?

 

[Skydiver]

Since you showed that your PacketReader derives from BinaryReader I'm going to assume that your PacketWriter derives from BinaryWriter, you could simply have the following:

class PacketWriter : BinaryWriter
{
:
    public void WriteMessage(string message)
    {
        // no need to write chunk size since BinaryWriter.Write(string) does that automatically
        Write(message);
    }
:
}

class PacketReader : BinaryReader
{
:
    public string ReadMessage()
    {
        // no need to read the chunk size since BinaryReader.ReadString() does that to determine how big the string will be
        return ReadString();
    }
:
}