On an ESP32S3 using ESP-IDF (not Arduino), and the azure-iot-middleware-freertos, specifically the sample_azure_iot_pnp.c, the ESP32S3-Device is sending telemetry data to the IoTHub, and I can view the data arriving, using the Azure IoT Explorer.
I have set up a route and a custom endpoint, to route to a NoSQL CosmosDB.
The messages are not routed, even though the displayed body in the Telemetry pane of the IoTExplorer is correct.
If I cut and paste this body's content to the Test Routes Body, then the route tests OK with the Query condition I have set.
I suspect it has to do with setting the ContentEncoding = "utf-8",
ContentType = "application/json".
Also making sure the message is encoded in utf-8.
The example code uses the following function to setup the telemetry data:
uint32_t ulCreateTelemetry( uint8_t * pucTelemetryData,
uint32_t ulTelemetryDataSize,
uint32_t * ulTelemetryDataLength )
{
int result = snprintf( ( char * ) pucTelemetryData, ulTelemetryDataSize,
myTELEMETRY_MESSAGE, myACTION); //sampleazureiotMESSAGE, xDeviceCurrentTemperature
if( ( result >= 0 ) && ( result < ulTelemetryDataSize ) )
{
*ulTelemetryDataLength = result;
result = 0;
}
else
{
result = 1;
}
return result;
}
The following is the calling function in a RTOS Task loop:
/* Publish messages with QoS1, send and process Keep alive messages. */
for( ; ; )
{
/* Hook for sending Telemetry */
if( ( ulCreateTelemetry( ucScratchBuffer, sizeof( ucScratchBuffer ), &ulScratchBufferLength ) == 0 ) &&
( ulScratchBufferLength > 0 ) )
{
xResult = AzureIoTHubClient_SendTelemetry( &xAzureIoTHubClient,
ucScratchBuffer, ulScratchBufferLength,
NULL, eAzureIoTHubMessageQoS1, NULL );
configASSERT( xResult == eAzureIoTSuccess );
}
/* Hook for sending update to reported properties */
ulReportedPropertiesUpdateLength = ulCreateReportedPropertiesUpdate( ucReportedPropertiesUpdate, sizeof( ucReportedPropertiesUpdate ) );
if( ulReportedPropertiesUpdateLength > 0 )
{
xResult = AzureIoTHubClient_SendPropertiesReported( &xAzureIoTHubClient, ucReportedPropertiesUpdate, ulReportedPropertiesUpdateLength, NULL );
configASSERT( xResult == eAzureIoTSuccess );
}
LogInfo( ( "Attempt to receive publish message from IoT Hub.\r\n" ) );
xResult = AzureIoTHubClient_ProcessLoop( &xAzureIoTHubClient,
sampleazureiotPROCESS_LOOP_TIMEOUT_MS );
configASSERT( xResult == eAzureIoTSuccess );
/* Leave Connection Idle for some time. */
LogInfo( ( "Keeping Connection Idle...\r\n\r\n" ) );
vTaskDelay( sampleazureiotDELAY_BETWEEN_PUBLISHES_TICKS );
}
The Telemetry Data is defined as follows:
#define myTELEMETRY_MESSAGE "{\"action\":\"%s\",\"ph_weight_device_mac\":\"XXXXXXXXXXXX\",\"ph_weight_index\":99,\"ph_weight_dev_nr\":0,\"ph_weight_block\":\"TEST\",\"ph_weight_activity\":\"ACTIVITY 2\",\"ph_weight_mode\":105,\"ph_weight_tag\":\"6D2517EBB4\",\"ph_weight_weight\":2.6202,\"ph_weight_date\":\"2022-12-07\",\"ph_weight_time\":\"13:45:12\"}"
The myACTION as follows:
#define myACTION "addPackhouseWeight"
The Azure IoT function that actually publishes the message:
AzureIoTResult_t AzureIoTHubClient_SendTelemetry( AzureIoTHubClient_t * pxAzureIoTHubClient,
const uint8_t * pucTelemetryData,
uint32_t ulTelemetryDataLength,
AzureIoTMessageProperties_t * pxProperties,
AzureIoTHubMessageQoS_t xQOS,
uint16_t * pusTelemetryPacketID )
{
AzureIoTMQTTResult_t xMQTTResult;
AzureIoTResult_t xResult;
AzureIoTMQTTPublishInfo_t xMQTTPublishInfo = { 0 };
uint16_t usPublishPacketIdentifier = 0;
size_t xTelemetryTopicLength;
az_result xCoreResult;
if( pxAzureIoTHubClient == NULL )
{
AZLogError( ( "AzureIoTHubClient_SendTelemetry failed: invalid argument" ) );
xResult = eAzureIoTErrorInvalidArgument;
}
else if( az_result_failed(
xCoreResult = az_iot_hub_client_telemetry_get_publish_topic( &pxAzureIoTHubClient->_internal.xAzureIoTHubClientCore,
( pxProperties != NULL ) ? &pxProperties->_internal.xProperties : NULL,
( char * ) pxAzureIoTHubClient->_internal.pucWorkingBuffer,
pxAzureIoTHubClient->_internal.ulWorkingBufferLength,
&xTelemetryTopicLength ) ) )
{
AZLogError( ( "Failed to get telemetry topic: core error=0x%08lx", xCoreResult ) );
xResult = AzureIoT_TranslateCoreError( xCoreResult );
}
else
{
xMQTTPublishInfo.xQOS = xQOS == eAzureIoTHubMessageQoS1 ? eAzureIoTMQTTQoS1 : eAzureIoTMQTTQoS0;
xMQTTPublishInfo.pcTopicName = pxAzureIoTHubClient->_internal.pucWorkingBuffer;
xMQTTPublishInfo.usTopicNameLength = ( uint16_t ) xTelemetryTopicLength;
xMQTTPublishInfo.pvPayload = ( const void * ) pucTelemetryData;
xMQTTPublishInfo.xPayloadLength = ulTelemetryDataLength;
/* Get a unique packet id. Not used if QOS is 0 */
if( xQOS == eAzureIoTHubMessageQoS1 )
{
usPublishPacketIdentifier = AzureIoTMQTT_GetPacketId( &( pxAzureIoTHubClient->_internal.xMQTTContext ) );
}
/* Send PUBLISH packet. */
if( ( xMQTTResult = AzureIoTMQTT_Publish( &( pxAzureIoTHubClient->_internal.xMQTTContext ),
&xMQTTPublishInfo, usPublishPacketIdentifier ) ) != eAzureIoTMQTTSuccess )
{
AZLogError( ( "Failed to publish telemetry: MQTT error=0x%08x", xMQTTResult ) );
xResult = eAzureIoTErrorPublishFailed;
}
else
{
if( ( xQOS == eAzureIoTHubMessageQoS1 ) && ( pusTelemetryPacketID != NULL ) )
{
*pusTelemetryPacketID = usPublishPacketIdentifier;
}
AZLogInfo( ( "Successfully sent telemetry message" ) );
xResult = eAzureIoTSuccess;
}
}
return xResult;
}
The data shows up in the Telemetry Pane of Azure IoT Explorer:
Fri Dec 09 2022 12:35:24 GMT+0200 (South Africa Standard Time):
{
"body": {
"action": "addPackhouseWeight",
"ph_weight_device_mac": "XXXXXXXXXXXX",
"ph_weight_index": 99,
"ph_weight_dev_nr": 0,
"ph_weight_block": "TEST",
"ph_weight_activity": "ACTIVITY 2",
"ph_weight_mode": 105,
"ph_weight_tag": "AD2512EBB4",
"ph_weight_weight": 2.6202,
"ph_weight_date": "2022-12-07",
"ph_weight_time": "13:45:12"
},
"enqueuedTime": "Fri Dec 09 2022 12:35:24 GMT+0200 (South Africa Standard Time)",
"systemProperties": {
"iothub-connection-device-id": "XXXXXXXXXXXX",
"iothub-connection-auth-method": "{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"iothub-connection-auth-generation-id": "638052575116976851",
"iothub-enqueuedtime": 1670582124804,
"iothub-message-source": "Telemetry",
"dt-dataschema": "dtmi:com:loadassist:Packhouse;2"
}
}
So, it is arriving, but probably Base64 encoded and Azure IoT Explorer just deals with it and displays properly.
But I suspect, that the routing is not working, as the example code does not set the encoding and the type of data as explained above. There are Arduino examples where they setup and encode a message, but nothing to that effect in this SDK.
I cannot see if they do it anywhere else in their example code.
I don't know how to go about this with the SDK, and the used MQTT WebSocket.
I would also like to set a property value, so that I can possibly query and route on that, and not have the "action": "addPackhousWeight" stored with the weight data to cosmosDB.
Well I hope I have been clear enough.
I tried a support Ticket with Microsoft, but it was closed by them and unanswered.
Thx
Comment Added:
I found the following in the sample_azure_iot.c not included in the sample_azure_iot_pnp.c :
/* Create a bag of properties for the telemetry */
xResult = AzureIoTMessage_PropertiesInit( &xPropertyBag, ucPropertyBuffer, 0, sizeof( ucPropertyBuffer ) );
configASSERT( xResult == eAzureIoTSuccess );
xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "content-encoding", sizeof( "content-encoding" ) - 1,
( uint8_t * ) "utf-8", sizeof( "utf-8" ) - 1 );
configASSERT( xResult == eAzureIoTSuccess );
xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "content-type", sizeof( "content-type" ) - 1,
( uint8_t * ) "application%2Fjson", sizeof( "application%2Fjson" ) - 1 );
configASSERT( xResult == eAzureIoTSuccess );
or as follows:
/* Create a bag of properties for the telemetry */
xResult = AzureIoTMessage_PropertiesInit( &xPropertyBag, ucPropertyBuffer, 0, sizeof( ucPropertyBuffer ) );
configASSERT( xResult == eAzureIoTSuccess );
xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "contentEncoding", sizeof( "contentEncoding" ) - 1,
( uint8_t * ) "utf-8", sizeof( "utf-8" ) - 1 );
configASSERT( xResult == eAzureIoTSuccess );
xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "contentType", sizeof( "contentType" ) - 1,
( uint8_t * ) "application%2Fjson", sizeof( "application%2Fjson" ) - 1 );
configASSERT( xResult == eAzureIoTSuccess );
Tried both content-type, and contentType etc.
But alas, still not routing ...
At last I got it to work:
xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "$.ce", sizeof( "$.ce" ) - 1,
( uint8_t * ) "utf-8", sizeof( "utf-8" ) - 1 );
configASSERT( xResult == eAzureIoTSuccess );
xResult = AzureIoTMessage_PropertiesAppend( &xPropertyBag, ( uint8_t * ) "$.ct", sizeof( "$.ct" ) - 1,
( uint8_t * ) "application%2Fjson", sizeof( "application%2Fjson" ) - 1 );
configASSERT( xResult == eAzureIoTSuccess );
I found the solution here:
https://azure.microsoft.com/sv-se/blog/iot-hub-message-routing-now-with-routing-on-message-body/
So adding "$.ce" and "$.ct" helped to get it to work. Literally took me days.
Hope it will help someone else.
I also learnt this:
/**
* #note The properties init API will not encode properties. In order to support
* the following characters, they must be percent-encoded (RFC3986) as follows:
* - `/` : `%2F`
* - `%` : `%25`
* - `#` : `%23`
* - `&` : `%26`
* Only these characters would have to be encoded. If you would like to avoid the need to
* encode the names/values, avoid using these characters in names and values.
*/
I'm trying to imitate http://threejs.org/examples/canvas_geometry_cube.html
via using a 3d.obj file with a texture mapping.
But i keep receiving the following error plus the rotation is totally off.
Uncaught TypeError: Cannot read property 'rotation' of undefined
Full demo is here http://wunderfauks.com/test/examples/test.html
Where you assign the value for group the code for loading has not executed yet and obj does not have a value. Move those 10 lines of code inside your loading function.
I've done it, if anyone needs to know you need to apply a matrix rotation.
Here's an example.
function onWindowResize() {
windowHalfX = window.innerWidth / 2;
windowHalfY = window.innerHeight / 2;
camera.aspect = window.innerWidth / window.innerHeight;
camera.updateProjectionMatrix();
renderer.setSize( window.innerWidth, window.innerHeight );
}
//
function onDocumentMouseDown( event ) {
event.preventDefault();
document.addEventListener( 'mousemove', onDocumentMouseMove, false );
document.addEventListener( 'mouseup', onDocumentMouseUp, false );
document.addEventListener( 'mouseout', onDocumentMouseOut, false );
mouseXOnMouseDown = event.clientX - windowHalfX;
targetRotationOnMouseDown = targetRotation;
}
function onDocumentMouseMove( event ) {
mouseX = event.clientX - windowHalfX;
targetRotation = targetRotationOnMouseDown + ( mouseX - mouseXOnMouseDown ) * 0.02;
}
function onDocumentMouseUp( event ) {
document.removeEventListener( 'mousemove', onDocumentMouseMove, false );
document.removeEventListener( 'mouseup', onDocumentMouseUp, false );
document.removeEventListener( 'mouseout', onDocumentMouseOut, false );
}
function onDocumentMouseOut( event ) {
document.removeEventListener( 'mousemove', onDocumentMouseMove, false );
document.removeEventListener( 'mouseup', onDocumentMouseUp, false );
document.removeEventListener( 'mouseout', onDocumentMouseOut, false );
}
function onDocumentTouchStart( event ) {
if ( event.touches.length === 1 ) {
event.preventDefault();
mouseXOnMouseDown = event.touches[ 0 ].pageX - windowHalfX;
targetRotationOnMouseDown = targetRotation;
}
}
function onDocumentTouchMove( event ) {
if ( event.touches.length === 1 ) {
event.preventDefault();
mouseX = event.touches[ 0 ].pageX - windowHalfX;
targetRotation = targetRotationOnMouseDown + ( mouseX - mouseXOnMouseDown ) * 0.05;
}
}
function animate() {
requestAnimationFrame( animate );
render();
stats.update();
}
function render() {
//plane.rotation.y = group.rotation.y += ( targetRotation - group.rotation.y ) * 0.05;
var rotation = new THREE.Matrix4().makeRotationY(Math.PI/2);
var translation = new THREE.Matrix4().makeTranslation(0, 0, 0);
rotation = group.rotation.y += (targetRotation - group.rotation.y);
var transformation = new THREE.Matrix4().makeTranslation(0, 0, 0).makeRotationY(rotation * 0.01 * 0.3) ;
group.applyMatrix(transformation);
renderer.render( scene, camera );
}
Here's the problem:
I wrote a program to receive data from a StockClient.
When I use DDESpy, I can see s message such as
Time:xxxx hwndTo=0xe00b0 Message(Posted)=Data:<br>
hwndFrom=0x1f06dc, status=2000(fRelease ) fmt=0x1("CF_TEXT")<br>
Data= "value"<br>
Item= "xxxxx"<br>
But my ddecallback function doesn't work (occasionally it can work), why?
DdeInitialize( &_dde_inst, dde_callback, APPCMD_CLIENTONLY/*|CBF_FAIL_ALLSVRXACTIONS*/, 0 );
HSZ _server = DdeCreateStringHandle ( _dde_inst, "DDEServer", CP_WINANSI );
if( !_server )
return -1;
HSZ _topic = DdeCreateStringHandle( _dde_inst, "TOPIC1", CP_WINANSI );
HCONV _conv = DdeConnect( _dde_inst, _server, _topic, NULL );
if( !_conv )
return -1;
for( int i = 0; i < ITEM_NUM; ++i )
{
HSZ _item = DdeCreateStringHandle( _dde_inst, _list[i], CP_WINANSI );
if( _item )
{
DWORD _result;
HDDEDATA _data = DdeClientTransaction( NULL, NULL, _conv, _item, CF_TEXT, XTYP_ADVSTART, TIMEOUT_ASYNC, &_result );
UINT _error = DdeGetLastError(_dde_inst );
DdeFreeStringHandle( _dde_inst, _item );
int _reserve = DMLERR_NO_ERROR;
if( _error != 0 )
{
fprintf( stdout, "dde error = %d\n", _error );
continue;
}
}
}
Wikipedia uses an "HTML sitemap" to link to every single content page. The huge amount of pages has to be split into lots of groups so that every page has a maximum of ca. 100 links, of course.
This is how Wikipedia does it:
Special: All pages
The whole list of articles is divided into several larger groups which are defined by their first and last word each:
"AAA rating" to "early adopter"
"earth" to "lamentation"
"low" to "priest"
...
When you click one single category, this range (e.g. "earth" to "lamentation") is divided likewise. This procedure is repeated until the current range includes only ca. 100 articles so that they can be displayed.
I really like this approach to link lists which minimizes the number of clicks needed to reach any article.
How can you create such an article list automatically?
So my question is how one could automatically create such an index page which allows clicks to smaller categories until the number of articles contained is small enough to display them.
Imagine an array of all article names is given, how would you start to program an index with automatical category-splitting?
Array('AAA rating', 'abdicate', ..., 'zero', 'zoo')
It would be great if you could help me. I don't need a perfect solution but a useful approach, of course. Thank you very much in advance!
Edit: Found the part in Wikipedia's software (MediaWiki) now:
<?php
/**
* Implements Special:Allpages
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
* http://www.gnu.org/copyleft/gpl.html
*
* #file
* #ingroup SpecialPage
*/
/**
* Implements Special:Allpages
*
* #ingroup SpecialPage
*/
class SpecialAllpages extends IncludableSpecialPage {
/**
* Maximum number of pages to show on single subpage.
*/
protected $maxPerPage = 345;
/**
* Maximum number of pages to show on single index subpage.
*/
protected $maxLineCount = 100;
/**
* Maximum number of chars to show for an entry.
*/
protected $maxPageLength = 70;
/**
* Determines, which message describes the input field 'nsfrom'.
*/
protected $nsfromMsg = 'allpagesfrom';
function __construct( $name = 'Allpages' ){
parent::__construct( $name );
}
/**
* Entry point : initialise variables and call subfunctions.
*
* #param $par String: becomes "FOO" when called like Special:Allpages/FOO (default NULL)
*/
function execute( $par ) {
global $wgRequest, $wgOut, $wgContLang;
$this->setHeaders();
$this->outputHeader();
$wgOut->allowClickjacking();
# GET values
$from = $wgRequest->getVal( 'from', null );
$to = $wgRequest->getVal( 'to', null );
$namespace = $wgRequest->getInt( 'namespace' );
$namespaces = $wgContLang->getNamespaces();
$wgOut->setPagetitle(
( $namespace > 0 && in_array( $namespace, array_keys( $namespaces) ) ) ?
wfMsg( 'allinnamespace', str_replace( '_', ' ', $namespaces[$namespace] ) ) :
wfMsg( 'allarticles' )
);
if( isset($par) ) {
$this->showChunk( $namespace, $par, $to );
} elseif( isset($from) && !isset($to) ) {
$this->showChunk( $namespace, $from, $to );
} else {
$this->showToplevel( $namespace, $from, $to );
}
}
/**
* HTML for the top form
*
* #param $namespace Integer: a namespace constant (default NS_MAIN).
* #param $from String: dbKey we are starting listing at.
* #param $to String: dbKey we are ending listing at.
*/
function namespaceForm( $namespace = NS_MAIN, $from = '', $to = '' ) {
global $wgScript;
$t = $this->getTitle();
$out = Xml::openElement( 'div', array( 'class' => 'namespaceoptions' ) );
$out .= Xml::openElement( 'form', array( 'method' => 'get', 'action' => $wgScript ) );
$out .= Html::hidden( 'title', $t->getPrefixedText() );
$out .= Xml::openElement( 'fieldset' );
$out .= Xml::element( 'legend', null, wfMsg( 'allpages' ) );
$out .= Xml::openElement( 'table', array( 'id' => 'nsselect', 'class' => 'allpages' ) );
$out .= "<tr>
<td class='mw-label'>" .
Xml::label( wfMsg( 'allpagesfrom' ), 'nsfrom' ) .
" </td>
<td class='mw-input'>" .
Xml::input( 'from', 30, str_replace('_',' ',$from), array( 'id' => 'nsfrom' ) ) .
" </td>
</tr>
<tr>
<td class='mw-label'>" .
Xml::label( wfMsg( 'allpagesto' ), 'nsto' ) .
" </td>
<td class='mw-input'>" .
Xml::input( 'to', 30, str_replace('_',' ',$to), array( 'id' => 'nsto' ) ) .
" </td>
</tr>
<tr>
<td class='mw-label'>" .
Xml::label( wfMsg( 'namespace' ), 'namespace' ) .
" </td>
<td class='mw-input'>" .
Xml::namespaceSelector( $namespace, null ) . ' ' .
Xml::submitButton( wfMsg( 'allpagessubmit' ) ) .
" </td>
</tr>";
$out .= Xml::closeElement( 'table' );
$out .= Xml::closeElement( 'fieldset' );
$out .= Xml::closeElement( 'form' );
$out .= Xml::closeElement( 'div' );
return $out;
}
/**
* #param $namespace Integer (default NS_MAIN)
* #param $from String: list all pages from this name
* #param $to String: list all pages to this name
*/
function showToplevel( $namespace = NS_MAIN, $from = '', $to = '' ) {
global $wgOut;
# TODO: Either make this *much* faster or cache the title index points
# in the querycache table.
$dbr = wfGetDB( DB_SLAVE );
$out = "";
$where = array( 'page_namespace' => $namespace );
$from = Title::makeTitleSafe( $namespace, $from );
$to = Title::makeTitleSafe( $namespace, $to );
$from = ( $from && $from->isLocal() ) ? $from->getDBkey() : null;
$to = ( $to && $to->isLocal() ) ? $to->getDBkey() : null;
if( isset($from) )
$where[] = 'page_title >= '.$dbr->addQuotes( $from );
if( isset($to) )
$where[] = 'page_title <= '.$dbr->addQuotes( $to );
global $wgMemc;
$key = wfMemcKey( 'allpages', 'ns', $namespace, $from, $to );
$lines = $wgMemc->get( $key );
$count = $dbr->estimateRowCount( 'page', '*', $where, __METHOD__ );
$maxPerSubpage = intval($count/$this->maxLineCount);
$maxPerSubpage = max($maxPerSubpage,$this->maxPerPage);
if( !is_array( $lines ) ) {
$options = array( 'LIMIT' => 1 );
$options['ORDER BY'] = 'page_title ASC';
$firstTitle = $dbr->selectField( 'page', 'page_title', $where, __METHOD__, $options );
$lastTitle = $firstTitle;
# This array is going to hold the page_titles in order.
$lines = array( $firstTitle );
# If we are going to show n rows, we need n+1 queries to find the relevant titles.
$done = false;
while( !$done ) {
// Fetch the last title of this chunk and the first of the next
$chunk = ( $lastTitle === false )
? array()
: array( 'page_title >= ' . $dbr->addQuotes( $lastTitle ) );
$res = $dbr->select( 'page', /* FROM */
'page_title', /* WHAT */
array_merge($where,$chunk),
__METHOD__,
array ('LIMIT' => 2, 'OFFSET' => $maxPerSubpage - 1, 'ORDER BY' => 'page_title ASC')
);
$s = $dbr->fetchObject( $res );
if( $s ) {
array_push( $lines, $s->page_title );
} else {
// Final chunk, but ended prematurely. Go back and find the end.
$endTitle = $dbr->selectField( 'page', 'MAX(page_title)',
array_merge($where,$chunk),
__METHOD__ );
array_push( $lines, $endTitle );
$done = true;
}
$s = $res->fetchObject();
if( $s ) {
array_push( $lines, $s->page_title );
$lastTitle = $s->page_title;
} else {
// This was a final chunk and ended exactly at the limit.
// Rare but convenient!
$done = true;
}
$res->free();
}
$wgMemc->add( $key, $lines, 3600 );
}
// If there are only two or less sections, don't even display them.
// Instead, display the first section directly.
if( count( $lines ) <= 2 ) {
if( !empty($lines) ) {
$this->showChunk( $namespace, $from, $to );
} else {
$wgOut->addHTML( $this->namespaceForm( $namespace, $from, $to ) );
}
return;
}
# At this point, $lines should contain an even number of elements.
$out .= Xml::openElement( 'table', array( 'class' => 'allpageslist' ) );
while( count ( $lines ) > 0 ) {
$inpoint = array_shift( $lines );
$outpoint = array_shift( $lines );
$out .= $this->showline( $inpoint, $outpoint, $namespace );
}
$out .= Xml::closeElement( 'table' );
$nsForm = $this->namespaceForm( $namespace, $from, $to );
# Is there more?
if( $this->including() ) {
$out2 = '';
} else {
if( isset($from) || isset($to) ) {
global $wgUser;
$out2 = Xml::openElement( 'table', array( 'class' => 'mw-allpages-table-form' ) ).
'<tr>
<td>' .
$nsForm .
'</td>
<td class="mw-allpages-nav">' .
$wgUser->getSkin()->link( $this->getTitle(), wfMsgHtml ( 'allpages' ),
array(), array(), 'known' ) .
"</td>
</tr>" .
Xml::closeElement( 'table' );
} else {
$out2 = $nsForm;
}
}
$wgOut->addHTML( $out2 . $out );
}
/**
* Show a line of "ABC to DEF" ranges of articles
*
* #param $inpoint String: lower limit of pagenames
* #param $outpoint String: upper limit of pagenames
* #param $namespace Integer (Default NS_MAIN)
*/
function showline( $inpoint, $outpoint, $namespace = NS_MAIN ) {
global $wgContLang;
$inpointf = htmlspecialchars( str_replace( '_', ' ', $inpoint ) );
$outpointf = htmlspecialchars( str_replace( '_', ' ', $outpoint ) );
// Don't let the length runaway
$inpointf = $wgContLang->truncate( $inpointf, $this->maxPageLength );
$outpointf = $wgContLang->truncate( $outpointf, $this->maxPageLength );
$queryparams = $namespace ? "namespace=$namespace&" : '';
$special = $this->getTitle();
$link = $special->escapeLocalUrl( $queryparams . 'from=' . urlencode($inpoint) . '&to=' . urlencode($outpoint) );
$out = wfMsgHtml( 'alphaindexline',
"$inpointf</td><td>",
"</td><td>$outpointf"
);
return '<tr><td class="mw-allpages-alphaindexline">' . $out . '</td></tr>';
}
/**
* #param $namespace Integer (Default NS_MAIN)
* #param $from String: list all pages from this name (default FALSE)
* #param $to String: list all pages to this name (default FALSE)
*/
function showChunk( $namespace = NS_MAIN, $from = false, $to = false ) {
global $wgOut, $wgUser, $wgContLang, $wgLang;
$sk = $wgUser->getSkin();
$fromList = $this->getNamespaceKeyAndText($namespace, $from);
$toList = $this->getNamespaceKeyAndText( $namespace, $to );
$namespaces = $wgContLang->getNamespaces();
$n = 0;
if ( !$fromList || !$toList ) {
$out = wfMsgWikiHtml( 'allpagesbadtitle' );
} elseif ( !in_array( $namespace, array_keys( $namespaces ) ) ) {
// Show errormessage and reset to NS_MAIN
$out = wfMsgExt( 'allpages-bad-ns', array( 'parseinline' ), $namespace );
$namespace = NS_MAIN;
} else {
list( $namespace, $fromKey, $from ) = $fromList;
list( , $toKey, $to ) = $toList;
$dbr = wfGetDB( DB_SLAVE );
$conds = array(
'page_namespace' => $namespace,
'page_title >= ' . $dbr->addQuotes( $fromKey )
);
if( $toKey !== "" ) {
$conds[] = 'page_title <= ' . $dbr->addQuotes( $toKey );
}
$res = $dbr->select( 'page',
array( 'page_namespace', 'page_title', 'page_is_redirect' ),
$conds,
__METHOD__,
array(
'ORDER BY' => 'page_title',
'LIMIT' => $this->maxPerPage + 1,
'USE INDEX' => 'name_title',
)
);
if( $res->numRows() > 0 ) {
$out = Xml::openElement( 'table', array( 'class' => 'mw-allpages-table-chunk' ) );
while( ( $n < $this->maxPerPage ) && ( $s = $res->fetchObject() ) ) {
$t = Title::makeTitle( $s->page_namespace, $s->page_title );
if( $t ) {
$link = ( $s->page_is_redirect ? '<div class="allpagesredirect">' : '' ) .
$sk->linkKnown( $t, htmlspecialchars( $t->getText() ) ) .
($s->page_is_redirect ? '</div>' : '' );
} else {
$link = '[[' . htmlspecialchars( $s->page_title ) . ']]';
}
if( $n % 3 == 0 ) {
$out .= '<tr>';
}
$out .= "<td style=\"width:33%\">$link</td>";
$n++;
if( $n % 3 == 0 ) {
$out .= "</tr>\n";
}
}
if( ($n % 3) != 0 ) {
$out .= "</tr>\n";
}
$out .= Xml::closeElement( 'table' );
} else {
$out = '';
}
}
if ( $this->including() ) {
$out2 = '';
} else {
if( $from == '' ) {
// First chunk; no previous link.
$prevTitle = null;
} else {
# Get the last title from previous chunk
$dbr = wfGetDB( DB_SLAVE );
$res_prev = $dbr->select(
'page',
'page_title',
array( 'page_namespace' => $namespace, 'page_title < '.$dbr->addQuotes($from) ),
__METHOD__,
array( 'ORDER BY' => 'page_title DESC',
'LIMIT' => $this->maxPerPage, 'OFFSET' => ($this->maxPerPage - 1 )
)
);
# Get first title of previous complete chunk
if( $dbr->numrows( $res_prev ) >= $this->maxPerPage ) {
$pt = $dbr->fetchObject( $res_prev );
$prevTitle = Title::makeTitle( $namespace, $pt->page_title );
} else {
# The previous chunk is not complete, need to link to the very first title
# available in the database
$options = array( 'LIMIT' => 1 );
if ( ! $dbr->implicitOrderby() ) {
$options['ORDER BY'] = 'page_title';
}
$reallyFirstPage_title = $dbr->selectField( 'page', 'page_title',
array( 'page_namespace' => $namespace ), __METHOD__, $options );
# Show the previous link if it s not the current requested chunk
if( $from != $reallyFirstPage_title ) {
$prevTitle = Title::makeTitle( $namespace, $reallyFirstPage_title );
} else {
$prevTitle = null;
}
}
}
$self = $this->getTitle();
$nsForm = $this->namespaceForm( $namespace, $from, $to );
$out2 = Xml::openElement( 'table', array( 'class' => 'mw-allpages-table-form' ) ).
'<tr>
<td>' .
$nsForm .
'</td>
<td class="mw-allpages-nav">' .
$sk->link( $self, wfMsgHtml ( 'allpages' ), array(), array(), 'known' );
# Do we put a previous link ?
if( isset( $prevTitle ) && $pt = $prevTitle->getText() ) {
$query = array( 'from' => $prevTitle->getText() );
if( $namespace )
$query['namespace'] = $namespace;
$prevLink = $sk->linkKnown(
$self,
htmlspecialchars( wfMsg( 'prevpage', $pt ) ),
array(),
$query
);
$out2 = $wgLang->pipeList( array( $out2, $prevLink ) );
}
if( $n == $this->maxPerPage && $s = $res->fetchObject() ) {
# $s is the first link of the next chunk
$t = Title::MakeTitle($namespace, $s->page_title);
$query = array( 'from' => $t->getText() );
if( $namespace )
$query['namespace'] = $namespace;
$nextLink = $sk->linkKnown(
$self,
htmlspecialchars( wfMsg( 'nextpage', $t->getText() ) ),
array(),
$query
);
$out2 = $wgLang->pipeList( array( $out2, $nextLink ) );
}
$out2 .= "</td></tr></table>";
}
$wgOut->addHTML( $out2 . $out );
if( isset($prevLink) or isset($nextLink) ) {
$wgOut->addHTML( '<hr /><p class="mw-allpages-nav">' );
if( isset( $prevLink ) ) {
$wgOut->addHTML( $prevLink );
}
if( isset( $prevLink ) && isset( $nextLink ) ) {
$wgOut->addHTML( wfMsgExt( 'pipe-separator' , 'escapenoentities' ) );
}
if( isset( $nextLink ) ) {
$wgOut->addHTML( $nextLink );
}
$wgOut->addHTML( '</p>' );
}
}
/**
* #param $ns Integer: the namespace of the article
* #param $text String: the name of the article
* #return array( int namespace, string dbkey, string pagename ) or NULL on error
* #static (sort of)
* #access private
*/
function getNamespaceKeyAndText($ns, $text) {
if ( $text == '' )
return array( $ns, '', '' ); # shortcut for common case
$t = Title::makeTitleSafe($ns, $text);
if ( $t && $t->isLocal() ) {
return array( $t->getNamespace(), $t->getDBkey(), $t->getText() );
} else if ( $t ) {
return null;
}
# try again, in case the problem was an empty pagename
$text = preg_replace('/(#|$)/', 'X$1', $text);
$t = Title::makeTitleSafe($ns, $text);
if ( $t && $t->isLocal() ) {
return array( $t->getNamespace(), '', '' );
} else {
return null;
}
}
}
Not a great approach as you don't have a way of stopping when you get to the end of the list. You only want to split the items if there is more items than your maximum (although you may want to add some flexibility there, as you could get to the stage where you have two items on a page).
I assume that the datasets would actually come from a database, but using your $items array for ease of display
At its simplest, assuming it is coming from a web page that is sending an index number of the start and end, and that you have checked that those numbers are valid and sanitised
$itemsPerPage = 50; // constant
$itemStep = ($end - $start) / $itemsPerPage;
if($itemStep < 1)
{
for($i = $start; $i < $end; $i++)
{
// display these as individual items
display_link($items[$i]);
}
}
else
{
for($i = $start; $i < $end; $i += $itemStep)
{
$to = $i + ($itemStep - 1); // find the end part
if($to > $end)
$to = $end;
display_to_from($items[$i], $items[$to]);
}
}
where the display functions display the links as you want. However, one of the problems doing it like that is that you may want to adjust the items per page, as you run the risk of having a set of (say) 51 and ending up with a link from 1 to 49, and another 50 to 51.
I don't understand why you are arranging it in groups in your pseudocode, as you are going from page to page doing further chops, so you only need the start and end of each section, until you get to the page where all the links will fit.
-- edit
The original was wrong. Now you divide the amount of items you have to go through by the maximum items you want to display. If it is 1000, this will be listing ever 20 items, if it is 100,000 it will be every 2,000. If it is less than the amount you show, you can show them all individually.
-- edit again - to add some more about the database
No, you are right, you don't want to load 2,000,000 data records, and you don't have to.
You have two options, you can make a prepared statement such as "select * from articles where article = ?" and loop through the results getting one at a time, or if you want to do it in one block - Assuming a mysql database and the code above,
$numberArray = "";
for($i = $start; $i < $end; $i += $itemStep)
{
$to = $i + ($itemStep - 1); // find the end part
if($to > $end)
$to = $end;
// display_to_from($items[$i], $items[$to]);
if( $i != $start)
$numberArray += ", ";
$numberArray.= $i.", ".$to;
}
$sqlQuery = "Select * from articles where article_id in (".$numberArray.")";
... do the mysql select and go through the results, using alternate rows as the start and end
This gives you a query like 'Select * from articles where article_id in (1,49,50,99,100,149... etc)'
The process that as a normal set
My approach in pseudo-code:
$items = array('air', 'automatic', 'ball', ..., 'yield', 'zero', 'zoo');
$itemCount = count($items);
$itemsPerPage = 50; // constant
$counter = 0;
foreach ($items as $item) {
$groupNumber = floor($counter/$itemsPerPage);
// assign $item to group $groupNumber
$counter++;
}
// repeat this procedure recursively for each of the new groups
Do you think this is a good approach? Can you improve or complete it?