I can't speak towards what makes a great library that 3rd parties are going to be using, but I can make a few points about what makes Lua code "clean" which is always a desired feature.
1. As kikito pointed out, libraries are generally considered a collection of things, and in that regard we use the dot notation rather than the colon notation for accessing that content. I do have one slight disagreement with this policy, and that's on the use of singletons. If you're designing a library that's going to act in the capacity of a singleton, then you
may want to consider the colon notation, to indicate that it's a very stateful thing that's being used. But you should totally avoid singletons, especially with 3rdparty libraries. To put it simply: your "Util" library containing unassociated functions, use the dot.
2. Limit scope of variables. The number of locals per scope are a fixed amount, and I forget the amount off hand, but it's always a good idea to keep variables local to the scope they're used in. On this point, the do...end scope keywords are your best friend. So if one particular function in your library is going to have a large closure around it, encase it in a do end block so that other functions don't have to share that scope.
3. Separation of concerns. This is a larger software engineering principle, but make sure your functions are really only doing one thing. If you see your function is twiddling bits and poking your widgets, they're probably better off as two functions.
4. Decide on Mechanism or Policy. Is your library provided a core mechanism? If so then you want it to really just be a collection of functions that return whatever their results are and shies away from keeping an internal state. Just look at how love's own built-in libraries work: The functions are designed to take as input the output of other functions. This contrasts with something like a Widget library, which is all about policy and is really just wrapping some other library's mechanisms.
5. Make efforts toward self-documentation. The difference between a variables named x1, x2, and x3 aren't apparent, but you know exactly what I mean when I say xStart, xEnd, and xStep. The commenting operator -- is all but missing from most Lua code because we actually don't need it, because we can do such a good job at naming things correctly.
6. Avoid "Code Smell", meaning sometimes code has to get a little messy, but don't give up on the whole file and allow it to get messy everywhere. Too many levels of indentation or too many vague variable names can really ruin code. To put it another way, code isn't done when it works, it's done when you've had a chance to rewrite it to make it look pretty and it still works.
7. Test. Test test test. Test test test test test. You can't know your code works until you've run it through a test. So if you write a library named foo.lua, you also have to write a test harness named testfoo.lua. It doesn't need to be complex, it can just be a list of assert statements that prove the output of your functions are what you say they are.
I'll leave it there, but there are tons more principles that you can read up on, of which my favorites are Eric Raymond's list:
http://en.wikipedia.org/wiki/Unix_philo ... ic_Raymond