In T246901#5974963, @Mrjohncummings wrote:

@Keegan do you know what kinds of programming skills/languages would be needed to implement this?

If you (or anyone) is actually interested in doing the technical work for this, feel free to reach out with any questions. I'm not offering to do it, but I am offering to answer any questions about how code base works, if anyone gets "stuck" trying to do it.

@Bawolff thats very helpful, thanks very much, is there anything you could add to the task description that would help people to get started/not get stuck as easily?

@brion worked on glTF support a while ago. Looks to me like it got quite far, maybe possible to rebase it and take a stab at it.

https://gerrit.wikimedia.org/r/c/3d2png/+/415495

@brion could you describe what the issues you ran into were and how people could help?

Thanks

As a note of support, I'd very much love to see support for glTF. There are so many amazing full colour models of NASA spacecraft, such as https://mars.nasa.gov/resources/24584/curiosity-rover-3d-model/

I have just participated in the Hack4OpenGLAM Hackathon, part of the Creative Commons Global Summit 2020. One of the datasets available was a set of 1,000+ CC0-licensed 3D models of cultural heritage available on Sketchfab.

So I dug into this content for a week, and my conclusion is: 3D is certainly booming in cultural heritage, and there's so much fantastic content out there now. It has grown so much in the past few years.
But also, many models are only interesting with textures, unfortunately. This one is a great example - it's the glazing on the dish that makes it worth looking at.

You can see a summary of what I did at the hackathon here: https://commons.wikimedia.org/wiki/File:3D_models_from_Sketchfab_and_Wikimedia%E2%80%99s_structured_data_-_-Hack4OpenGLAM_at_CC_Summit_2020.pdf

FYI, in response to my hackathon project I was approached on Twitter by developers of the Modelviewer software, who suggested it as a tool to make integration of textured 3D models in currently widely supported file formats very easy for us.

I've created a separate ticket for that proposal: T266379: Use Modelviewer as 3D viewer for Wikimedia projects

In T246901#6575897, @Spinster wrote:

FYI, in response to my hackathon project I was approached on Twitter by developers of the Modelviewer software, who suggested it as a tool to make integration of textured 3D models in currently widely supported file formats very easy for us.

I've created a separate ticket for that proposal: T266379: Use Modelviewer as 3D viewer for Wikimedia projects

This would be really amazing, any idea how the process would work to use existing software?

@Spinster : "most models are only interesting with textures."

I could not agree more! Thank you for sharing your hackerthon summary and also looking into this. I suspect that one we have the ability to showcase textured 3D objects it will become a very important component of many Wikipedia articles.

For dynamic media like videos and 3D files, two different pieces of software are required. One is the thumbnailer, which turns the dynamic media into a static image. That's currently done by rTDTP 3d2png. The second is the client-side dynamic viewer, which allows users to view and interact with the dynamic content. That's currently implemented by Extension:3D as a part of MediaViewer. Both depend on https://threejs.org to do the actual heavy lifting of rendering 3D files with WebGL. Using the same library in both applications is important to reduce the visual difference between the static and dynamic versions and so that editors have a stable platform to work with.

As far as the process for implementing a third-party-developed viewer, the most recent example would probably be TimedMediaHandler's switch from Kaltura to VideoJS: T100106: Replace Kaltura player with Video.js. Extending MediaViewer like the current solution is likely to be the best option, if it's decided that switching renderers is a good idea. That is, however, likely to be more work than adapting the existing viewer.

@John_Cummings @Spinster I'm keen to see how this is coming. Is there an on-wiki discussion about it?

@Sj I'm sorry I missed this before, I don't think so, what would be the goal of having an on wiki discussion about it?

What would be a good path to achieving this? Who might possibly be interested in working on this and what would be the process to implement it?

From the GLAM mailing list [1]:

"Last week, the Wikimedia Foundation shared its draft annual plan on Meta and invited input via open calls in different time zones. You might have noticed that Maryana recognized that Commons is in desperate need of repair and made it a priority for the year ahead.

"There will be two calls dedicated to Commons:

"Wikimedia Commons 1 on Tuesday 26 April at 10:00 to 11:30 UTC

"Wikimedia Commons 2 on Thursday 28 April at 16:00 to 17:30 UTC"

Maybe this could be raised on one or other of those calls?

[1] https://lists.wikimedia.org/hyperkitty/list/glam@lists.wikimedia.org/thread/EJ262OLQUPBFL2HNOINYIGX5X65QAWJQ/

Just a quick heads up. I am seriously considering putting in a presentation application to talk about this subject (the need for 3D support on Commons/Wikipedia) at this year's Wikimania. I just wanted to know if anyone here is interested in joining me on that application? I feel even more strongly now on this issue than I did back in 2020 when I first commented on this issue.

Hello! I will be in Wikimania, I have other presentation plans, but I can help you with this, if there's appetite.

In T246901#8692485, @Theklan wrote:

Hello! I will be in Wikimania, I have other presentation plans, but I can help you with this, if there's appetite.

Thanks Theklan, that would be most welcome. Would you like to chat some time about it? The presentation plan at this stage is really very simple; here is the need for 3D support on Wiki and how it can be used, here are the advancements in displaying 3D objects else where, lets give developers support to work on this, Q&A and discussions.

I´m also interested in this and would like to help, I have examples of how could be used.
I would also like to chat with you.

@Discott you can contact me via Telegram, for sure. I'm thinking of people working on GLAM also. @GabrielLucas welcome!

I created this telegram group to talk about it
https://t.me/+tMgoJxx8D7I5NTQ8

@Discott @GabrielLucas @Theklan @John_Cummings nice to see! I'd be glad to be part of a workshop on supporting new formats, the overarching needs / costs / reasons-for-stalling are quite similar. And I think the best approach to realize this [since there is a necessary WM-core bottleneck] is to define a time-delimited project to identify and add a range of formats [and then update the workflow for doing so, ping interested people, &c].

Otherwise each subgroup wastes hundreds of hours working through the same bits of red tape.

There are some interesting other applications, beside sketchfab to look at:

• DFG Viewer Demo Example Implementation funded by DFG (German Research Foundation)
• kompakkt funded by NFDI4Culture (German research )

Just to get some inspiration for a UI

Is this also related T277930?

Here is a proposal on Commons with quite a lot of interest in the topic: https://commons.wikimedia.org/wiki/Commons:Village_pump/Proposals#Enable_textured_3D_files_on_Commons

In T246901#9054129, @Kristbaum wrote:

Here is a proposal on Commons with quite a lot of interest in the topic: https://commons.wikimedia.org/wiki/Commons:Village_pump/Proposals#Enable_textured_3D_files_on_Commons

The discussion was closed successfully with: "Broad consensus for allowing this."

What should be the next step?

Would be great if some epic task could be created that tracks all more of the actual implementation tasks for different file formats, most of which haven't moved forward for some time e.g T184803 or T100081. And of course adding it to some higher priority queue would be very appreciated.

I suppose T44725 is too all-encompassing for what you have in mind?

We do need some tractable way to divide and conquer this issue: it's obvious to some people that we should support a much wider range of formats (as that would bring in more users, use cases, devs caring about those formats) but the Uncertainty and Doubt around what's involved in integrating a new format is diffusing that energy -- even when, as in this case, it's a large source of new knowledge, connected to many other collaborative communities, and supported on Commons.

1. @Aklapper you proposed closing T44725 some time back as being too large / vague / undirected compared to other options; how do you see this family of requests and threads being better organized?
2. If the WMF doesn't have anyone responsible for currently prioritizing adding new formats, can we turn these ~20 steps into more of a checklist that anyone can make progress on for a given format or family of formats, reducing the core-dev decisions to a security review + performance review + integration after all the rest has been sorted?
   1. In that case can the checklist for a particular format reside on a wiki? Phab doesn't quite seem ideal.

@Sj: I likely do not have a good answer. All I can say is that I believe that a ticket which covers anything related to any file formats related to what individuals may interpret into the term (multi)media is an extremely wide scope these days. :) Your mileage may vary, of course...

Did we already examine what file type(s) are most suitable for textured meshes?

Then we can figure out what we have to take care of when implementing and what viewer is recommended.

For the point cloud files, I made a suggestion for LAS as filetype (T100081).

Note that improving media file format support is still not something that's explicitly receiving resourcing at Wikimedia Foundation, so it's good to have a low cost in terms of implementation work (note that this includes thumbnailing via thumbor, an entirely separate package), low risk in terms of tooling safety from libraries and executables used in rendering on the server or client side, an easy way to embed the resulting view into mobile apps without too much platform-specific coding, and an easy safe way to downgrade support if security problems are found in the future.

Remember this needs to be done with no dedicated team for file format or image/media embedding or maintenance of that code and infrastructure.

[Note: in general pulling in binary file parsers/renderers is a big extra security risk for server-side execution. Pulling in a large JavaScript rendering library is also a security risk for client-side safety, as it could expose an XSS injection vector in the library that can be used against the entire page. So there's a relatively high bar, and we've seen problems in production for real with Graphs etc.]

So if we add server-side rendering, we need a clean way to sandbox it, that works both in standalone installs and via Thumbor, and can be relatively sensibly managed and packaged, and made disableable with a switch-flip. I'm also hoping to work on some iframe sandboxing stuff for media embedding, which will later be usable in stuff like media renderers (which we can port the existing video & 3d interfaces into when it's ready)... I think we've got some tasks open for that related to Graphs, I'll track that down later :D

A few thoughts:

1. In all major JavaScript WebGL libraries (three.js, Babylon.js, PlayCanvas, ...) the preferred model format for display is glTF 2.0. It is lightweight compared to alternatives; supports meshes, lines, and point clouds; provides more modern/advanced material appearance definitions; does not require binary or proprietary parsers; and offers web-friendly compression options. Some compression methods would require a WASM decoder.
2. Other formats might be preferable for users to upload. In particular, OBJ, FBX, STL, and USDZ are common, but less ideal for rendering in a lightweight client.
3. I suspect the proposed model HTML element would be immensely helpful in reducing the security and performance footprint of this feature, compared to a WebGL library. The current state of the proposal focuses on support for USDZ and glTF models. If something were needed sooner, https://modelviewer.dev/ is comparable.

Disclaimer: I'm a maintainer of three.js and a contributor to the glTF file format specification.

gITF sounds fair to me. It is so common that this filetype is offered on different websites and is supported in several 3D software titles.

I tested it with a model of a wheel rim, it does work well. But it seems like the gITF file needs a dependent binary file, right?

Otherwise I think we can add this filetype to Commons, maybe with other filetypes later.

Greetings

There are two main container conventions for glTF files:

• .glb files are binary, and self-contained, with no external resources. The binary includes (1) a file header, (2) a JSON chunk describing the contents, and (3) a binary chunk containing data for geometry, textures, and perhaps animations.
• .gltf files are JSON, and typically contain references (by relative path) to one or more external resources. Those resources might include one or more .bin files with geometry and animation data, and images used for textures.

Those are conventions, not rules – it is possible to construct a .glb with external resources, or a .gltf without any. But I think it would be very reasonable to enforce a rule that users may upload only the .glb container, and that it cannot contain references to external resources. This is the most common use of the format.

Facebook previously supported uploading 3D models in posts, though they've since removed that feature. In case it's helpful as a reference, these were their rules and restrictions on user uploads:

• https://web.archive.org/web/20190903182539/https://developers.facebook.com/docs/sharing/3d-posts/asset-requirements
• https://web.archive.org/web/20190903221309/https://developers.facebook.com/docs/sharing/3d-posts/glb-tutorials

Summary:

• We support GLB files (GLB is the binary version of the glTF 2.0 file format).
• Your GLB must be under 3MB in total. The smaller the better.
• Strip out unused data whenever possible. Unused vertex colors, UV's, etc. all add to the file size.
• Use PNG or JPEG files for textures.
• Use JPEG files unless you need transparency. This will make your files much smaller.
• Textures must be power-of-2 in each dimension (512x1024 for example).
• Textures can be up to 4k, but try to keep them under 2k.
• Once you have a GLB file, you can just drag it into a new post, pick a background color and share it.
• The PBR shader can use these textures:
  • Base Color (rgba, with alpha optionally driving transparency)
  • Normal
  • Emissive
  • ORM (Packed into channels in a single texture: Red has the Occlusion texture, Green has the Roughness texture and Blue has the Metalness texture.)
• Additionally, the PBR shader will use vertex color, if present.

The power-of-two size requirement on textures is there to benefit low-end devices supporting WebGL 1.0 but not WebGL 2.0. Support for WebGL 2.0 is more common today than it was then, but still missing on 3-5% of devices according to different surveys:

• https://caniuse.com/webgl2
• https://web3dsurvey.com/webgl2

Not listed in formal requirements, models were also required to pass validation (excluding some low-severity hints) under the Khronos Group's official validation library.

Is the gITF format as such suitable to be implemented in Wikimedia Commons with its attributes as mentioned above? :)

What is the current state/progress? :)

@PantheraLeo1359531 See everything that is written in this task before

I’ll be blunt. No one is working on this, no one is planned to be working on this. There is no magic pixie dust to move it forward. It needs ppl spending time to program it and no one has volunteered to do so, so nothing will change until that changes.

Asking for what the progress is, is always completely useless, the ticket should and does reflect the state/progress accurately.

Maybe we can acquire one or more voluntary people?

Maybe it is a stupid question, but I want to figure out what we can achieve, because it would be a fantastic addition to Commons.

If you have the ability, feel free to volunteer :) [seriously, if anyone is interested and has some background knowledge in programming and wants to do this, we will provide advice and help]

If you want to recruit someone, probably better to look elsewhere, as everyone on this task is already aware of the request.

I am afraid, I can't, but I will keep my eyes open :)

Would be really nice if we can achieve this

The WMF agreed to do this in principle [so long as it did not use too much in the way of developer time] at the last Wikimania. Perhaps I should reach out to the WMF and remind them of this commitment?

I wrote a short report on my chat with the WMF at the Commons village pump which can be read here; https://commons.wikimedia.org/wiki/Commons:Village_pump/Proposals/Archive/2023/09#Enable_textured_3D_files_on_Commons

Please do!

I also asked some people if they know someone who could contribute.

In T246901#9226554, @brion wrote:

So if we add server-side rendering, we need a clean way to sandbox it, that works both in standalone installs and via Thumbor, and can be relatively sensibly managed and packaged, and made disableable with a switch-flip.

@Legoktm on SJ's computer here :) We can use Shellbox for sandboxing, which transparently handles standalone installs, but someone will need to do the Thumbor integration. I already ported all the deployed MediaHandlers to use Shellbox so there's plenty of examples from others to copy from!

@Discott

Did WMF answer already?

In T246901#9324384, @Sj wrote:

In T246901#9226554, @brion wrote:

So if we add server-side rendering, we need a clean way to sandbox it, that works both in standalone installs and via Thumbor, and can be relatively sensibly managed and packaged, and made disableable with a switch-flip.

@Legoktm on SJ's computer here :) We can use Shellbox for sandboxing, which transparently handles standalone installs, but someone will need to do the Thumbor integration. I already ported all the deployed MediaHandlers to use Shellbox so there's plenty of examples from others to copy from!

Hey, do you have an idea of how to progress this? It would be such a good addition and this is getting quite a vintage phab ticket. Is there a way to add it to someone's list of work at WMF to investigate further and plan the next steps?

Thanks

@brion what do you say?

In T246901#9394722, @Sj wrote:

@brion what do you say?

Probably the wrong account: @bvibber

Still waiting on WMF to budget time to look into doing more multimedia work. :)

Please note there is as far as I know no active Wikimedia-sponsored work at present on any multimedia file format support except for self-driven 20% time by me plus volunteer work by @TheDJ on TimedMediaHandler (audio/video playback).

Problem analysis has not changed since my initial reply years ago:

• use GLTF format for color models, it's standard and well supported
• update the frontend and the backend renderers
• if necessary, replace them with newer libraries

One additional thing: Safari has native USD model support, depending on whether there are format patent issues or not (needs investigation) it might be worth including support for native display via a conversion. This would be a stretch goal, potentially assignable via Mobile Apps if interested for iOS/iPad/Vision Pro compatibility.

(I see this is also in the Hackathon workboard -- perfect timing! I'll have updates on the patch for folks to test by then but I'll be remote for it. :D)

In T246901#9723607, @bvibber wrote:

Still waiting on WMF to budget time to look into doing more multimedia work. :)

Please note there is as far as I know no active Wikimedia-sponsored work at present on any multimedia file format support except for self-driven 20% time by me plus volunteer work by @TheDJ on TimedMediaHandler (audio/video playback).

I asked a staff member weeks ago and she said fiscal year ends at the end of June. Maybe then the budget can be used for the implementation, among others? :)