[SOLVED]Downloading large files + display status

[shinprog]

Hello,

I use in my game large files (20Mb+),and I would like to make it download files.
I found a code using socket,but I want to display in the game window the download status (for example,a progress bar with the percentage and the download speed).I can do it in python but not in lua :/
Have you any solution ?

Thanks you <3 !

[0x25a0]

Heya!

What you're asking for is not simple, but it can be done It comes down to three problems:

1. Normally, http.request doesn't give you any updates about the download progress; you just call it and it returns when the download is done (or when it ran into an error). What you need is a way to get updated on the download progress while the download is happening. For this, you can write a custom sink. The sink that you pass to http.request receives the downloaded data in chunks. Normally these chunks would then be written to a file. By writing a custom sink, you can run some code whenever a new chunk of data arrives. A sink is just a function that accepts two arguments: a chunk of data, and possibly an error message.

Here is a sink that prints the size of each chunk whenever it receives a new chunk:

Code: Select all

local function progress_sink(chunk, err)
  if chunk then
    print(string.format("received %d bytes", #chunk))
  elseif err then
    print(err)
  else
    print("Download finished")
  end
end

To use this sink, pass it to the http.request function:

Code: Select all

local request = {
    url = "https://love2d.org/style/screenshots/oh-my-giraffe.png",
    method = "GET",
    sink = custom_sink,
}
local success, status_code = http.request(request)

When you run this, you will get a lot of prints saying "received X bytes". That's a step in the right direction.
However, now the downloaded file is not saved anywhere. To change that, we can combine our custom sink with one of the standard sinks that are included in the lua-socket library:

Code: Select all

-- this function creates a sink that forwards the downloaded chunk to another sink
local function progress_sink(output_sink)
  -- this function is the actual sink
  return function(chunk, err)
    if chunk then
      print(string.format("received %d bytes", #chunk))
    else

      -- if chunk is nil, then the download is finished, or we have an error
      if err then
        print(err)
      else
        print("Download finished")
      end

    end

    -- forward chunk and err to the underlying sink
    return output_sink(chunk, err)
  end
end

-- this is the sink that will write the downloaded data to a file
local output_sink = ltn12.sink.file(io.open("test.png", "wb"))

local request = {
  url = url,
  sink = progress_sink(output_sink),
  method = "GET",
}

local success, status_code = http.request(request)

Now, while the file is downloading, our progress sink will keep us updated on the number of bytes that have been downloaded.
However, there is still another problem: the http.request function still blocks while the download is running. This means that you won't be able to draw anything to the screen until the download is finished. This brings us to the second problem:

2. You need to move the download to a separate thread. When you start the download on a separate thread, then you can continue to draw things to the screen while the download is running in the background. You can use love2d's Channels to send data back and forth between the two threads. For example, you can have one channel on which download URLs are sent to the downloader thread, and another channel that is used by the downloader thread to send back progress updates.

3. Finally, to display a percentage you need to know the file size of the thing you're downloading. That is stored in the response header as "content-length". However, http.request only returns this header after the download is finished. But to display a nice progress bar, you need that information right away. The solution here is to send two requests; the first one only requests the header, and the second request actually downloads the file.

Attached is an example project to show how all of this can be done. It's public domain, so use it however you want

I hope this helps. I'll let you work out how to display the download speed
Let me know if you have any questions.

*Edit: The demo should now run on Windows, too.
*Edit: Updated the demo to fix thread error handling and wrong usage of io.open
*Edit: The previous version was buggy, as Beelz rightfully pointed out. I've rewritten part of the demo to avoid Lua's io module

[brogrammer]

there are webservers that don't support the header request
most do but some special/customized servers don't

[shinprog]

Thanks you for you replies !!
The code works,it's excellent ! I'll edit when i can do it.

you should publish it in a single topic for others
[post solved]

there are webservers that don't support the header request
most do but some special/customized servers don't

In my webiste,it work

[shinprog]

Hello,
I need your help
It works in command line but when i package-it in a .love file,it not works...
any suggestions ?

[0x25a0]

Ah whoops, that was my fault.
I forgot that you can't write to arbitrary files. I updated the demo. The fix involved two things:
- setting the identity of your game with `love.filesystem.setIdentity("name of your game")`
- saving the file to the save directory, instead of to the current directory, by passing a full filepath to the downloader.

In your project, instead of writing to a file inside the .love archive, you will need to write to a file within the save directory.
This seems to work:

Code: Select all

function love.load()
  --timer = 0
  default = love.graphics.newFont(14)
  title = love.graphics.newFont("resources/fonts/font02.ttf",40)
  text = love.graphics.newFont("resources/fonts/font01.ttf",20)

  download.load()
  love.filesystem.setIdentity("kagerou")
  if not love.filesystem.exists("resources/video/intro.ogv") then
    love.filesystem.createDirectory("resources/video")
    download.addqueue("http://kagescan.esy.es/fangame/realtime/version2/output.ogv",
                      love.filesystem.getSaveDirectory() .. "/resources/video/intro.ogv",
                      "Final !!\nDownloading intro...")
  end
  download.start()

  menu = 0
end

The downloaded file will then be stored within the save directory. You can see on https://love2d.org/wiki/love.filesystem where that is for your operating system.
Let me know if you have any further questions

[Beelz]

*Edit: Updated the demo to fix thread error handling and wrong usage of io.open

It's generally frowned upon to use Lua's IO functions directly. I got an error with your demo and had to change line 25 in main.lua:

Code: Select all

-- From
io.output(log_filename)
-- To
love.filesystem.write(log_filename, "")

[zorg]

*Edit: Updated the demo to fix thread error handling and wrong usage of io.open

It's generally frowned upon to use Lua's IO functions directly. I got an error with your demo and had to change line 25 in main.lua:

Not really frowned upon, it's just that you can't guarantee that it will work; paths work differently on different OS-es.
(Also, PhysFS (what Löve uses) does lack some functionality that lua's io has, and that's also true vice-versa.)

[0x25a0]

It's generally frowned upon to use Lua's IO functions directly. I got an error with your demo and had to change line 25 in main.lua:

Code: Select all

-- From
io.output(log_filename)
-- To
love.filesystem.write(log_filename, "")

Thanks for letting me know! Admittedly I didn't test that version as much as I should have, and made some naïve assumptions. The current version hopefully fixes this bug once and for all.
For the record, io.output defines where stdout goes; it's not the Lua equivalent of love.filesystem.write. But your change was incidentally a work-around for the bug that caused the error in the first place

I do think that there are legitimate cases where using Lua's io module makes more sense, or where love.filesystem is simply not sufficient.
But yeah, this might not be one of those cases. I've rewritten part of the demo to avoid Lua's io module (in particular, I had to re-implement ltn12's file sink, see downloader.lua:12), and that did simplify other parts of the demo, and would have avoided the error that you ran into.

[shinprog]

Thanks you a lot for your lövely help !!
It seems to works very well !!!!

[SOLVED]