Generic Method which returns either MemoryStream or Byte array

[Sinh_R]

Hi,

I need some guidance regarding the problem I am facing.

PROBLEM:
I am currently working on MinIO object storage (similar to Azure Blob and AWS S3) and I am currently trying to implement a method which will download the files uploaded in my local cloud storage.

For that I want to create a Class which will has 3 properties:

1. StatusCode: To return the status code (http codes)
2. Message: A brief regarding the status of the request
3. Content: This will be either of type MemoryStream or byte array and will have the actual content of the file. If fails then it will be null.

In order to handle this I decided to create a class something like this:

public class GetObjectResponse<T>
{
    public int StatusCode { get; set; }
    public string StatusMessage { get; set; }
    public T? Response { get; set; }
}

After this now I decided to create a method which will be called to download the object. This will only use one argument which will be the name of the file to be downloaded from the local cloud storage.
For this I decided to create a method which will be of generic type and came with some thing like this:

public async Task<GetObjectResponse<T>> GetObject<T>(string objectName)
{
    try
    {
        var connString = _minioConfig.Value.ConnectionString;
        var bucketName = _minioConfig.Value.BucketName;

        var client = CreateClient(connString);

        var downloadStream = new MemoryStream();

        if (!await client.BucketExistsAsync(new BucketExistsArgs().WithBucket(bucketName)))
        {
            throw new BucketNotFoundException(bucketName, "Bucket does not exists");
        }

        await client.GetObjectAsync
            (
            new GetObjectArgs()
            .WithBucket(bucketName)
            .WithObject(objectName)
            .WithCallbackStream(stream =>
                                {
                                    stream.CopyTo(downloadStream);
                                })
        );

        //If T is of type MemoryStream return GetObjectResponse<MemoryStream>
        //Else GetObjectResponse<byte[]>
    }
    catch (MinioException mex)
    {
        return new GetObjectResponse<T>
        {
            StatusCode = 400,
            StatusMessage = mex.Message,
            Response = default
        };
    }
    catch (Exception ex)
    {
        return new GetObjectResponse<T>
        {
            StatusCode = 400,
            StatusMessage = ex.Message,
            Response = default
        };
    }
}

Now the main problem is that I am not able to return the object of type MemoryStream. If I tried this on line no 28

return new GetObjectResponse<MemoryStream>
{
    //setting the properties
}

This is giving error saying cannot convert MemoryStream to T and doing that explicitly might be wrong.

So I need help on what can be done here?
Guidance in any way will be appreciated.

 

[Skydiver]

It looks like you are simply using generics to save yourself some typing. Since you only have two possible types, just implement two different versions of GetObject(). One that returns GetObjectResponse<MemoryStream>, and another that returns GetObjectResponse<byte[]>.

The main issue here is that you are trying to treat C# generics like C++ templates. Although the syntax may look the same, the two are completely different animals. The error that the computer is giving you is accurate. It has no way knowing how to convert MemoryStream to T. What is the compiler supposed to do if T is a double? How will it make a floating point number into a memory stream?

 

[Sinh_R]

It looks like you are simply using generics to save yourself some typing. Since you only have two possible types, just implement two different versions of GetObject(). One that returns GetObjectResponse<MemoryStream>, and another that returns GetObjectResponse<byte[]>.

The main issue here is that you are trying to treat C# generics like C++ templates. Although the syntax may look the same, the two are completely different animals. The error that the computer is giving you is accurate. It has no way knowing how to convert MemoryStream to T. What is the compiler supposed to do if T is a double? How will it make a floating point number into a memory stream?

Yes, the main purpose of using generics was to avoid writing the same logic again. As the complete logic for GetObject will be same except of setting the Response for either MemoryStream or byte[].
So it will be like I am repeating my logic twice which I really dont want to.

Still I will be working on the same creating 2 new methods for respective types and keeping the other logic in a different method(Like calling a method separate for getting the object).
If you or anyone can find out some work around and is willing to post it I will be greatfull.
Thanks for the advice.

 

[Skydiver]

If you are really trying to save on typing, then do one implementation that returns a byte array. Then declare another version that returns a memory stream and all it does is call the first implementation and creates a memory stream using the byte array returned by a call to the first version.

 

[cjard]

Mmm.. I wouldn't do it. You aren't giving callers the choice to avoid buffering a whole file into memory by providing a byte[] or MemoryStream. Worse, if they call ToArray() on the MS, the file will end up buffered twice

I'd go the other way, do all the logic in a method that returns a Stream (not a memorystream) and return the network stream; don't buffer it. Where the caller knows the file is huuuge and doesn't want it all in memory they can stream it. Perhaps you can also provide a convenience method that returns a byte array and CopyTo an array after you've sized it according to the file size but remember that you're working with storage utilities like S3/BlobSto here, that could reasonably hold files much larger than can be fit into an array/memory

The whole idea of streaming is to shuffle bytes from place1 to place2 via the local machine without burning up too many resources. Shuffling them from place1 down into the memory of the local machine, before shuffling them on is a decision that should be handed to the caller of your method rather than something you enforce

 

[Skydiver]

+1 about concerns about moving copies of the stream contents around and potential size of the stream.

As an aside, the reason why I started off with the byte array is that I know that the .NET Framework MemoryStream doesn't shuffle bytes around from place to place when you use the constructor that takes a byte array. It just takes a reference to the byte array.

Reference Source

public MemoryStream(byte[] buffer)
    : this(buffer, true) {
}
      
public MemoryStream(byte[] buffer, bool writable) {
    if (buffer == null) throw new ArgumentNullException("buffer", Environment.GetResourceString("ArgumentNull_Buffer"));
    :
    _buffer = buffer;
    :
}

Anyway, following along with that logic that people should just hold on to the low level Stream reference that comes from the API (as opposed to copying the data into byte buffer or another stream), then that means that people should also just hang on to IEnumerable<T> and IQueryable<T> objects returned by various API and quit insisting on calling ToList(). That will keep from having to send the returns of database queries across the network until it is really being used. And what if the query returns is bigger than what fits into memory? Who cares if the underlying DbContext or SqlConnection is closed, or the data changes and the database doesn't provide ACID guarantees, right?