View source for File Upload and Media Handling

{{toc numerate=1}}

===environment===
php.ini
  * upload_max_filesize
  * post_max_size
  * max_input_time
  * max_execution_time
  * max_file_uploads
  * session.gc_maxlifetime

storage
  1. file system
    2. global
    3. perpage
  2. database

====settings====
options
  1. ##upload##
    2. [0, 1, Admins] - refactor, last value is actually just the allowed user group
  2. ##upload_max_size##
  3. ##upload_quota##
  4. ##upload_quota_per_user##
  5. ##upload_banned_exts##
  6. ##upload_images_only##

  1. ##check_mimetype##
  2. ##img_create_thumbnail##
  3. ##img_max_thumb_width##

constants
  1. ##define('THUMB_DIR',						'file/thumb');##
  1. ##define('UPLOAD_GLOBAL_DIR',				'file/global');##
  1. ##define('UPLOAD_PER_PAGE_DIR',			'file/perpage');##

roles

permissions
  * $this->has_access('upload')

Anti-Hack
  Download Flood Minutes:
    The number of minutes to set flood control to. 
  Download Flood Control:
    The number of downloads allowed in period of 'Download Flood Minutes'.

====chmod====
Write access for files folders.

  1. User can read, write and execute
  1. Group can read and execute
  1. World can read and execute

Execute privilege on directories is required to access files inside them. (AT&T)

The important thing is that the webserver has write access.

Thus if the files directory is owned by the webserver user ( e.g. www-data ) ##755## would suffice. 
If the webserver is not owner, but is in the group owning the directory, ##775## would be better. 
##777## is the extreme case where all users on the system have full access. In most cases this would not be advisable.

====MIME type and file extensions====
https://en.wikipedia.org/wiki/Media_type
https://mimesniff.spec.whatwg.org/

Blacklisting File Extensions

Whitelisting File Extensions
%%
# Allowed uploadable file extensions and mimetypes

$extension_map	= [
	// application
	'bz'		=> ['application', 'application/x-bzip'],
	'bz2'		=> ['application', 'application/x-bzip2'],
	'doc'		=> ['application', 'application/msword'],
	'exe'		=> ['application', 'application/octet-stream'],
	'latex'		=> ['application', 'application/x-latex'],
	'gtar'		=> ['application', 'application/x-gtar'],
	'gz'		=> ['application', 'application/x-gzip'],
	'gzip'		=> ['application', 'application/x-gzip'],
	'pdf'		=> ['application', 'application/pdf'],
	'ppt'		=> ['application', 'application/mspowerpoint'],
	'ps'		=> ['application', 'application/postscript'],
	'tar'		=> ['application', 'application/x-tar'],
	'tgz'		=> ['application', 'application/x-compressed'],
	'torrent'	=> ['application', 'application/x-bittorrent'],
	'xls'		=> ['application', 'application/excel'],
	'zip'		=> ['application', 'application/x-zip-compressed'],
	'7z'		=> ['application', 'application/x-7z-compressed'],

	'odc'		=> ['application', 'application/vnd.oasis.opendocument.chart'],
	'odb'		=> ['application', 'application/vnd.oasis.opendocument.database'],
	'odf'		=> ['application', 'application/vnd.oasis.opendocument.formula'],
	'odg'		=> ['application', 'application/vnd.oasis.opendocument.graphics'],
	'odi'		=> ['application', 'application/vnd.oasis.opendocument.image'],
	'odp'		=> ['application', 'application/vnd.oasis.opendocument.presentation'],
	'ods'		=> ['application', 'application/vnd.oasis.opendocument.spreadsheet'],
	'odt'		=> ['application', 'application/vnd.oasis.opendocument.text'],


	// (legacy)
	'pptx'		=> ['application', 'application/vnd.openxmlformats-officedocument.presentationml.presentation'],
	'xlsx'		=> ['application', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'],
	'docx'		=> ['application', 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'],

	// audio
	'm4a'		=> ['audio', 'audio/mp4'],
	'mp3'		=> ['audio', 'audio/x-mpeg-3'],
	'ogg'		=> ['audio', 'audio/ogg'],
	'opus'		=> ['audio', 'audio/ogg'],

	// image
	'avif'		=> ['image', 'image/avif'],
	'gif'		=> ['image', 'image/gif'],
	'ico'		=> ['image', 'image/x-icon'],
	'jpe'		=> ['image', 'image/jpeg'],
	'jpeg'		=> ['image', 'image/jpeg'],
	'jpg'		=> ['image', 'image/jpeg'],
	'jxl'		=> ['image', 'image/jxl'],
	'png'		=> ['image', 'image/png'],
	'svg'		=> ['image', 'image/svg+xml'],
	'webp'		=> ['image', 'image/webp'],

	// text
	'conf'		=> ['text', 'text/plain'],
	'css'		=> ['text', 'text/css'],
	'htm'		=> ['text', 'text/html'],
	'html'		=> ['text', 'text/html'],
	'rtf'		=> ['text', 'text/richtext'],
	'txt'		=> ['text', 'text/plain'],
	'xml'		=> ['text', 'text/xml'],

	// video
	'mp4'		=> ['video', 'video/mp4'],
	'ogv'		=> ['video', 'video/ogg'],
	'webm'		=> ['video', 'video/webm'],
];
%%

"Content-Type" Header Validation

====HTML5====

form
  * ((https://www.w3.org/TR/html5/forms.html#attr-input-accept accept attribute))

====File APIs====

  * ((https://www.w3.org/TR/file-upload/ File API specification))
  * ((https://www.html5rocks.com/en/tutorials/file/dndfiles/ Reading files in JavaScript using the File APIs))

Libs
  * http://filedropjs.org/
  * http://resumablejs.com/
  * https://www.dropzonejs.com/

====Fetch API====

Uploading files using 'fetch' and 'FormData'
  * https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#Uploading_a_file
  * ((https://www.taniarascia.com/how-to-upload-files-to-a-server-with-plain-javascript-and-php/ How to Upload Files to a Server with Plain JavaScript and PHP))

====PHP====
((https://www.php.net/manual/en/features.file-upload.post-method.php POST method uploads))

$_FILES
  * ##$_FILES['uploadedfile']['name']##: The original name of the file on the client machine
  * ##$_FILES['uploadedfile']['type']##: The mime type of the file
  * ##$_FILES['uploadedfile']['size']##: The size of the file in bytes
  * ##$_FILES['uploadedfile']['tmp_name']##: The temporary filename in which the uploaded file was stored on the server.


functions
  1. mime_content_type
  2. pathinfo
  3. hash_file

====Image processing====

  1. GD Graphics Library
  2. ImageMagick

check the availability
%%
if (extension_loaded('gd'))
{
    print_r(gd_info());
}
else
{
    echo 'GD is not available.';
}

if (extension_loaded('imagick'))
{
    $imagick = new Imagick();
    print_r($imagick->queryFormats());
}
else
{
    echo 'ImageMagick is not available.';
}
%% 

===WackoWiki file model===

====File name sanitization====
All the control characters and Unicode ones should be removed from the filenames and their extensions without any exception. Also, the special characters such as "##;##", "##:##", "##>##", "##<##", "##/##" ,"##\##", additional "##.##", "##*##", "##%##", "##$##", and so on should be discarded as well. If it is applicable and there is no need to have Unicode characters, it is highly recommended to only accept Alpha-Numeric characters and only 1 dot as an input for the file name and the extension; in which the file name and also the extension should not be empty at all (regular expression: ##[a-zA-Z0-9]{1,200}\.[a-zA-Z0-9]{1,10}##).

%%
// prepare for translit
$name	= str_replace(['@', '%20', '+'], '-', $name);
$name	= preg_replace('/[\r\n\t -]+/', '_', $name);
$name	= trim($name, ' .-_');

// here would be place for translit
$t_name = $this->format($name, 'translit');
%% 

  1. global
    1. [file_name . extension]
    2. UPLOAD_GLOBAL_DIR
    1. direct file access
  1. local 
    2. [@ . page_id . @ . file_name . extension]
    3. UPLOAD_PER_PAGE_DIR
    3. file access via file hadler
      4. checks page ACLs against user permissions

===Database===
  1. file_name
  2. author (user_id)
  3. file_description
  4. caption
  5. views
  6. uploaded_dt
  7. modified_dt
  8. credit
  9. exif
  10. context use (file_link)
  11. page_id
  12. categories
  13. access, viewing privacy
  14. file_lang
  15. file_size
  16. picture_w
  17. picture_h
  18. file_ext
  19. mime_type

===Workflow===

validation
  1. extension
  2. mime type
  3. size

header for file handler
%%
"Content-Disposition: Attachment"
"X-Content-Type-Options: nosniff" 
%% 

  1. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition
  2. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options

database
  1. mimetype (127+1+127 = **255**)
https://stackoverflow.com/questions/643690/maximum-mimetype-length-when-storing-type-in-db

actions
  1. files
  2. upload

handler
  1. upload
  2. file
  3. attachments
  4. filemeta

functions
  *  wacko -> link
  * wackoformatter

hacks


ideas

===File Usage===
file:/lotus_flower.jpg?caption

The file syntax is processed in the ##link## function. Not to mention that the link function requires a refactoring.

In R5.5 a cation field was added to the file table.
%%(html)
<figure>
  <img src="https://wackowiki.org/doc/file/global/lotus_flower.jpg" alt="Lotus Flower" title="Lotus Flower (105 KiB)" width="640" height="427">
  <figcaption>Shinobazu Pond at Ueno Park at Ueno, Taito-ku in Tokyo. Original by <a href="https://www.flickr.com/photos/yoshikazut/">Yoshikazu TAKADA</a></figcaption>
</figure>
%%
Make the width of ##<figcaption>## match the width of the ##<img>## inside its ##<figure>## tag
%%(css)
figure { display: table; }
figcaption { display: table-caption; caption-side: bottom; }
%%

The file can be linked directly (file handler) or to filemeta handler.

Arguments can be added via ##?## and ##&##

##~file:/image.png?200x400&direct&caption##

====Supported Media====

#|
*| Media | Format |*
|| audio | m4a, mp3, ogg, opus ||
|| image | avif, gif, jpg, jpe, jpeg, jxl, png, svg, webp ||
|| video | mp4, ogv, webm ||
|#

  * https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types
  * https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Audio_codecs
  * https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs

=====Audio=====
##~file:/audio.opus?caption##
file:/kimiko_ishizaka___bach___well_tempered_clavier_book_1___01_prelude_no._1_in_c_major_bwv_846.ogg?caption

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/audio

=====Video=====
##~file:/video.webm?caption##
file:/41265046264_720p.mp4?caption

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video
https://developer.mozilla.org/en-US/docs/Web/Apps/Fundamentals/Audio_and_video_delivery/Adding_captions_and_subtitles_to_HTML5_video

====Embedding====
##~file:image.png##
##~file:audio.opus##
##~file:video.mp4##

====Media Parameters====

#|
*| | img | audio | video | notes |*
|| linking  ||
|| direct | x | o | o | ||
|| nolink | x | o | o | ||
|| linkonly | x | o | o | ||
|| meta | x | o | o | ||
|| alignment ||
|| right | x | x | x | ||
|| left | x | x | x | ||
|| center | x | x | x | ||
|| resizing  ||
|| 400x200 | x | o | x | ||
|| 600 | x | x | x | ||
|| others  ||
|| caption | x | x | x | ||
|| clear | x | x | x | ||
|#

=====Alignment=====
##~file:image.png?right##

##~file:image.png?right&clear##

=====Resizing=====
##~file:image.png?20x50##

file:/lotus_flower.jpg?caption&400

=====Linking=====
##~file:image.png?direct##
##~file:image.png?nolink##
##~file:image.png?linkonly##

By default, clicking on an image should brings up the ##filemeta## handler page.

How we manage to link an image to another page with the file: syntax?
  * ##""((file:/image.png some text))""##
  * ##""((/Page/Link file:/image.png))""##

=====Caching=====
##~file:image.png?nocache##

=====Others=====
##~file:image.png?caption##

+ audio and video options

===Bad Behavior strict mode  issue ===
bb_settings.conf
%%
strict = true
%%
status_key: 7ad04a8a

HTTP load failed with status 400. Load of media resource ~https://wackowiki.org/doc/file/global/41265046264_720p.mp4 failed.
HTTP load failed with status 400. Load of media resource ~https://wackowiki.org/doc/file/global/kimiko_ishizaka___bach___well_tempered_clavier_book_1___01_prelude_no._1_in_c_major_bwv_846.ogg failed.

[400] The automated program you are using is not permitted to access this server. Please use a different program or a standard Web browser.
-> Prohibited header 'Range' present

%%
Range: bytes=0-
Referer: https://wackowiki.org/doc/Dev/Release/R5.5/FileUpload
Accept-Language: en-US,en;q=0.5
Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
Host: wackowiki.org
Mod-Rewrite: on
Mod-Env: on
%% 
Suppose you have a video or audio of 4M bytes, when your browser request for video the first time it will send headers like this
%%
Host:localhost
Range:bytes=0- %%

Range header ##bytes=0-## means browser is asking server to return till whatever it can return ie. no end position is specified

https://bad-behavior.ioerror.us/support/faq/

===Search===

search
  1. file name
  2. description
  3. title
  4. cation

sorting
  1. size
  2. name
  3. date
  4. type

filter
  1. lang
  2. category
  3. user
  4. location


===Links===
  1. http://www.php.net/manual/en/features.file-upload.php
  1. https://www.owasp.org/index.php/Unrestricted_File_Upload
  1. https://www.acunetix.com/websitesecurity/upload-forms-threat/
  1. https://en.wikipedia.org/wiki/Media_type
  1. https://www.dokuwiki.org/images